John C. Whitehead
Department of Economics
Appalachian State University
Boone, North Carolina
September 20, 2016
Abstract. Desvousges, Mathews and Train (2015) find that their CVM survey does not pass the adding up test. Visual inspection suggests little difference in the pattern of data across subsamples and that the data suffers from non-monotonicity. Probit models estimating the effect of the bid on the yes responses with the lowest bid removed finds little evidence that the data passes the most basic validity test over 89% of the range of bids. The study may suffer from small samples, inexpensive online panels and poor survey design.
Key Words: Contingent Valuation, Adding up test, Dichotomous Choice Data, Non-monotonicity
JEL: Q51
Note: The author thanks Bill Desvousges for providing the data.
Introduction
Desvousges, Mathews and Train (2012) critique Chapman et al. (2009) and argue that contingent valuation method (CVM) studies must pass the “adding up test” to demonstrate adequate responsiveness to scope [see also Whitehead (2016), Chapman et al. (2016) and Desvousges, Mathews and Train (2016)]. Desvousges, Mathews and Train (DMT, 2015) field the Chapman et al. (2009) survey with new sample data and additional scenarios. They find that their CVM survey does not pass the adding up test.
Data
The DMT data for the five samples is presented in Table 1 and displayed in Figure 1. DMT argue that willingness to pay for the whole should be equal to willingness to pay for the sum of four parts (first, second, third and fourth). Visual inspection suggests that there is little difference in the pattern of data across subsamples except for the second sample which lies below each of the other bid curves at every bid except $125. Dichotomous choice contingent valuation surveys present respondents with a bid (i.e., cost) and ask them if they would be willing to pay the bid. As the bid increases the probability that respondents will agree to pay the bid tends to fall (Dickinson and Whitehead 2015). Visual inspection suggests that the DMT data suffers from non-monotonicity (i.e., the percentage does not always decrease as the bid increases).
Most of the curvature in the data appears to be as the bid rises from $10 to $45. The difference in the percentage of yes responses (D = %yes[$10] - %yes[$45]) is 5, 16, 21, 37 and 29 for the whole, first, second, third and fourth samples with an average of 22. The average difference for the other bid gradients are -4 (as the bid increases from $45 to $80), 5 ($80 to $125), 11 ($125 to $205) and 2 ($205 to$450). This last small difference illustrates the fat tails problem found by Parsons and Myers (2016).
Results
Probit models estimating the effect of the bid on the yes responses with the $10 bid removed are in Table 2. Only two of the five coefficients (B) on the bid variable are statistically significant at the p=.10 level. [the corrigendum is here: http://www.env-econ.net/2016/10/a-comment-on-an-adding-up-test-on-contingent-valuations-of-river-and-lake-quality-correction.html]
Conclusions
The DMT data does not pass the adding up test. It is also not clear if it passes the most basic validity test in contingent valuation over 89% of the range of bids. This conclusion is similar to DMT’s Table 3 and accompanying discussion (i.e., why did I feel the need to conduct this analysis? IDK!). DMT suggest that there is something wrong with the CVM. It may also be the case that there is something wrong with the DMT study.
DMT employ small sample sizes and data from a relatively inexpensive online panel. Also, DMT use the Chapman et al. (2009) survey for the whole and second versions of the study. These are two scenarios that perform relatively well statistically -- the whole and second scenarios pass the scope test. For the first, third and fourth scenarios DMT "modified its instruments as little as possible to represent these situations." Two of the three scenarios constructed by DMT perform poorly on the bid test. The one scenario that passes has a 100% larger sample size. It is difficult to know what is in these survey scenarios as they have not been shared as readily as the data.
Critiquing the CVM because it does not pass the adding up test when the bid curves are not altogether downward sloping is similar to criticizing a basketball player who can’t make a layup only when they fail to hit a three pointer. The player should really be encouraged to work on their layups.
Appendix
Comment on Desvousges, Mathews and Train (2015) continued: cognitive burden (Sept 21, 2016)
Comment on Desvousges, Mathews and Train (2015) continued: cost (Sept 22, 2016)
Comment on Desvousges, Mathews and Train (2015) continued: theory (Sept 24, 2016)
Comment on Desvousges, Mathews and Train (2015) continued: WTP Estimates (Sept 26, 2016)
Figure 1. Dichotomous Choice Bid Curves with DMT Data Exhibiting Non-Monotonicity
References
Chapman, David, Richard Bishop, Michael Hanemann, Barbara Kanninen, Jon Krosnick, Edward Morey and Roger Tourangeau. 2009. Natural Resource Damages Associated with Aesthetic and Ecosystem Injuries to Oklahoma’s Illinois River System and Tenkiller Lake.
Chapman, David J., Richard C. Bishop, W. Michael Hanemann, Barbara J. Kanninen, Jon A. Krosnick, Edward R. Morey, and Roger Tourangeau. "On the adequacy of scope test results: Comments on Desvousges, Mathews, and Train." Ecological Economics 130 (2016): 356-360. http://www.sciencedirect.com/science/article/pii/S0921800916306139
Desvousges, William, Kristy Mathews, and Kenneth Train. “An Adding Up Test on Contingent Valuations of River and Lake Quality.” Land Economics 91(2015): 556-571. http://le.uwpress.org/content/91/3/556.refs
Desvousges, William, Kristy Mathews, and Kenneth Train. “Reply to ‘On the adequacy of scope test results: Comments on Desvousges, Mathews, and Train’,” Ecological Economics 130 (2016): 361–362. http://www.sciencedirect.com/science/article/pii/S0921800916306139
Dickinson, David L., and John C. Whitehead. "Dubious and Dubiouser: Contingent Valuation and the Time of Day." Economic Inquiry 53, no. 2 (2015): 1396-1400. http://onlinelibrary.wiley.com/doi/10.1111/ecin.12161/full
Parsons, George R., and Kelley Myers. "Fat tails and truncated bids in contingent valuation: An application to an endangered shorebird species." Ecological Economics 129 (2016): 210-219. http://www.sciencedirect.com/science/article/pii/S0921800915301567
Whitehead, John C. "Plausible responsiveness to scope in contingent valuation." Ecological Economics 128 (2016a); 17-22. http://www.sciencedirect.com/science/article/pii/S0921800916302890
