Desvousges, Mathews and Train (2015) [DMT] find that their contingent valuation method (CVM) survey does not pass the adding up test. In a previous post (Whitehead, Sept. 20, 2016) I examined the data used by Desvousges, Mathews and Train (2015) and find that "it is ... not clear if [the data] passes the most basic validity test in contingent valuation over 89% of the range of bids."
There are other things wrong with the paper. DMT take the opportunity to make another comment on Haab et al. (2013) in the context of their empirical study. Here is the introduction to their section III titled "potential difficulties in implementing the adding-up test":
There are several potential difficulties that must be addressed in implementing an adding-up test. Haab et al. (2013) describe these issues and seem to suggest that the potential problems are so great that they outweigh the potential benefits of the test. We believe that these issues need to be considered on a case-by-case basis. In the paragraphs below, we describe these potential difficulties and how they are addressed in our application.
There are four issues: cognitive burden, income effects, provision mechanism and cost. Here is what DMT says about cognitive burden:
The test requires that one part of the package of benefits be valued by respondents who are told that they already received another part. In many situations, this type of conditioning can be difficult for respondents to understand. In our application, we have been able to avoid this potential difficulty. One of the reasons we chose the Chapman et al. study is that its design is amenable to descriptions of incremental parts. As discussed below, the surveys for the incremental parts are the same from the respondents’ perspective as the survey for the whole. No additional cognitive burden is imposed. In the original study for the base program (the whole), the years in which recovery will occur with and without the proposed intervention were stated to respondents. We simply changed these stated years for each of the incremental parts. In fact, this change in stated years was used in the original study for differentiating its scope and base versions, which gave us the idea that other increments could be defined similarly. In other applications, describing increments might be more difficult. But it can be useful to identify studies, like Chapman et al.’s, in which the increments can be described without undue additional burden, and to apply adding-up tests in these applications.
I requested the surveys used by DMT so that I could examine them for face validity but only received an explanation that they were simple modifications of the Chapman et al. survey. I don't doubt that the descriptions of the good to be valued introduce no additional cognitive burden as stated in the paragraph above. The problem, in my opinion, is describing the counterfactual situation that one of the incremental parts has already been paid for by the respondent and and provided to the respondent. This description is necessary to account for income and substitution (or complementary, since there are more than 2 goods in the survey respondent's utility function) effects. For n goods, the adding up test requires n + 1 scenarios [1]:
- Elicit the willingness to pay for good A
- Elicit the willingness to pay for good B [after describing that good A has been provided at a cost of $X to the respondent]
- Elicit the willingness to pay for goods A and B
The adding up test is that WTP(1) + WTP(2) = WTP(3). The bracketed part of the second scenario is what introduces the additional cognitive burden. It is a counterfactual that exacerbates the difficulties of CVM (hypothetical bias, etc). A counterfactual situation increases the mental effort (i.e., cognitive burden) that can't be ignored even with the confident assertion that: "No additional cognitive burden is imposed." It is not clear that DMT actually tried to describe the counterfactual in part 2 since they rely on a statistical simulation to determine if income effects matter. So, it might be true that no additional cognitive burden is imposed in their survey. But, in that case DMT (2015) does not conduct an adding up test and there is no reason to think that WTP(1) + WTP(2) = WTP(3).
If I were given plenty of money to try to get the CVM to pass this test I would do it iteratively. First, I'd conduct two versions of a survey and elicit WTP for A and B to see just what those are without the counterfactual [a second version of this initial survey would elicit WTP for goods A and B and I'd conduct a scope test to determine the plausibility of the WTP difference (Whitehead 2016)].
In a second round of surveys I'd establish the budget constraint (not income) that respondents have available to spend on things like A and B and then in separate subsamples:
- Elicit the willingness to pay for good A (and see if the WTP is temporally reliable)
- Elicit the willingness to pay for good B [after describing that good A has been provided at a cost of $X to the respondent, where $X was determined from the initial survey and the respondent's budget for things like A and B is $X lower than they thought]
- Elicit the willingness to pay for goods A and B
That might work, i.e., the CVM might pass the adding up test, but something similar has already failed in laboratory experiments (Bateman et al. 1997).
Note:
[1] Desvousges, Mathews and Train's (2012) interpretation of Chapman et al. (2009) as an adding up test is, therefore, wrong (Whitehead 2016).
References
Bateman, Ian, Alistair Munro, Bruce Rhodes, Chris Starmer, and Robert Sugden. "Does part–whole bias exist? An experimental investigation." The Economic Journal 107, no. 441 (1997): 322-332.
Chapman, David, Richard Bishop, Michael Hanemann, Barbara Kanninen, Jon Krosnick, Edward Morey and Roger Tourangeau. 2009. Natural Resource Damages Associated with Aesthetic and Ecosystem Injuries to Oklahoma’s Illinois River System and Tenkiller Lake.
Desvousges, William, Kristy Mathews, and Kenneth Train. "Adequate responsiveness to scope in contingent valuation." Ecological Economics 84 (2012): 121-128. http://www.sciencedirect.com/science/article/pii/S0921800912003813
Desvousges, William, Kristy Mathews, and Kenneth Train. “An Adding Up Test on Contingent Valuations of River and Lake Quality.” Land Economics 91(2015): 556-571. http://le.uwpress.org/content/91/3/556.refs
Whitehead, John C. "Plausible responsiveness to scope in contingent valuation." Ecological Economics 128 (2016); 17-22. http://www.sciencedirect.com/science/article/pii/S0921800916302890
Whitehead, John C., "A comment on '“An Adding Up Test on Contingent Valuations of River and Lake Quality,'” The Environmental Economics Blog, September 20, 2016. http://www.env-econ.net/2016/09/a-comment-on-an-adding-up-test-on-contingent-valuations-of-river-and-lake-quality.html
