Environmental Economics

Reply to "Reply to Whitehead" by Desvousges, Mathews and Train (the correction)

As described in the introduction of my (draft) "Reply to 'Reply to Whitehead'", I suspect that I have used the incorrect confidence intervals when analyzing the Desvousges, Mathews and Train (2015) data. Park, Loomis and Creel (1991) introduced the Krinsky-Robb approach for estimating confidence intervals for willingness to pay estimates from dichotomous choice contingent valuation models. Cameron (1991) introduced the Delta Method approch. As indicated by their Google Scholar citations, 461 and 229 respectively, they have both been used extensively in the applied CVM literature. Hole (2007) compares the two approaches (along with Fieller and bootstrap approaches) and finds little difference in the approaches for well-behaved simulated data. However, Hole (2007) points out that the Delta Method requires that the willingness to pay be normally distributed for the confidence interval to be accurate. He states that "... it is likely that WTP is approximately normally distributed when the model is estimated using a large sample and the estimate of the coefficient for the cost attribute is sufficiently precise." (p. 830) In Whitehead (2020) I used the Delta Method confidence intervals in my statistical tests. This is very likely an inappropriate approach due to the imprecision of the estimate of the parameter on the cost amount. 

When working on Whitehead (2020) I used NLogit (www.limdep.com) software to estimate the confidence intervals. NLogit allows for both the Delta Method and Krinsky-Robb approaches to be used. But the Krinsky-Robb confidence intervals may require the assumption of normality. Hole (2007): "The [Krinsky-Robb] confidence interval could also be derived by using the draws to calculate the variance of WTP ..., but this approach, like the delta method confidence interval, hinges on the assumption that WTP is symmetrically distributed." (p. 831) Almost all of the Krinsky-Robb confidence intervals estimated by NLogit "blew up" when using the DMT (2015) data, in other words the upper and lower limits were in the 10s and 100s of thousands (positive and negative). This made little sense to me at the time but now my guess is that when the WTP normality assumption is violated the NLogit software can not handle the estimation. Typically, Delta Method and Krinsky-Robb confidence intervals are not very different when estimated in NLogit (as shown below). 

Following my reading of Desvousges, Mathews and Train (forthcoming) I thought through the above (obviously, I should have thought through it before) and estimated WTP using with the Krinsky-Robb intervals in SAS (my program is available upon request). My Krinsky-Robb intervals are akin to what Hole (2007) calls the Monte Carlo percentile approach. I take one million draws from the variance-covariance matrix and trip the α/2 highest and lowest WTP values, where α=0.05. Hole's (2007) Krinsky-Robb intervals are based on a resampling approach, but he finds little difference in the resampling and Monte Carlo Krinsky-Robb intervals.

For this analysis I am only using the whole scenario from DMT (2015) since this is sufficient to show that WTP for the whole can not be statistically distinguished from WTP for the sum of the parts with the Krinsky-Robb Monte Carlo percentile intervals. The logit models are presented below for the full sample (n=172), the sample with observations with missing demographics deleted (n=163) and the Chapman et al. (2009) data. In each model the constant and the coefficient on the cost amount are statistically different from zero. But, the precision of the cost coefficients with the DMT (2015) data are low relative to other CVM studies. Combined with small samples, the Desvousges, Mathews and Train WTP estimate may not be normally distributed. The Chapman et al. (2009) study, on the other hand, has a large sample size and a precisely estimated coefficient on the cost amount. 

The WTP estimates (restricting WTP to be positive) and confidence intervals are presented below. The Delta Method confidence intervals are estimated in NLogit and the Krinsky-Robb percentile intervals are estimated in SAS. The appropriateness of the Delta Method with the DMT (2015) data is questionnable. First, the Krinsky-Robb lower bound on the DMT (2015) full sample (n=172) estimate is less than 50% of the Krinsky-Robb lower bound. Second, the Krinsky-Robb upper bound is 269% larger than the Delta Method upper bound. The imprecision of the coefficient on the cost amount is driving the asymmetry. The cost estimate in the less than full sample (n=163) is estimated even more imprecisely. The Krinsky-Robb confidence interval includes zero. 

These results should be considered in contrast to the WTP estimate from the Chapman et al. (2009) data. The Delta Method and Krinsky-Robb intervals are very close. The symmetric Krinsky-Robb confidence interval estimated in NLogit is [236.90, 320.37] which is also very close to the Delta Method. One benchmark for symmetric confidence intervals in CVM studies, therefore, is a sample size greater than 1000 and a t-statistic on the coefficient for the cost coefficient of -9.5. Of course, sensitivity around these benchmarks should be assessed since not many CVM studies have these characteristics. (note: I'll do some of this sort of work when I go back to my past and analyze some of the CVM data from the old days that Desvousages, Mathews and Train (forthcoming) assert is just as bad as their own data.)

The point estimate of the sum of the WTP parts for the full sample is $1114.36. The WTP for the sum of the parts is within the Krinsky-Robb interval for the whole suggesting that we can not reject the hypothesis that WTP for the whole is equal to WTP for the sum of the parts at the p<0.05 level. The 90% interval is [264.84, 1314.14] which indicates that any statistical equality is at a confidence level below p=0.10. The point estimate of the sum of the WTP parts for the trimmed sample is $1079.73. Again, the WTP for the sum of the parts is within the Krinsky-Robb 95% interval for the whole. Note that the WTP for the whole is not different from zero with this sample so any statistical inference makes less sense than if WTP was different from zero. These results are consistent with my (erroneous) conclusion (Whitehead 2020) that the data in Desvousges, Mathews and Train (2015) are not sufficient to conclude that contingent valuation does not pass the adding up test. 

References

Cameron, Trudy Ann. "Interval estimates of non-market resource values from referendum contingent valuation surveys." Land Economics 67, no. 4 (1991): 413-421.

Hole, Arne Risa. "A comparison of approaches to estimating confidence intervals for willingness to pay measures." Health economics 16, no. 8 (2007): 827-840.

Park, Timothy, John B. Loomis, and Michael Creel. "Confidence intervals for evaluating benefits estimates from dichotomous choice contingent valuation studies." Land economics 67, no. 1 (1991): 64-73.

Reply to "Reply to Whitehead" by Desvousges, Mathews and Train (an introduction)

Desvousges, Mathews and Train (Land Economics, 2015) use the contingent valuation method (CVM) to conduct an adding-up test (i.e., does WTP_A + WTP_B = WTP_A+B?). They use the nonparametric Turnbull estimator and find that the data do not pass the adding-up test. This suggests that the CVM lacks internal validity.

In September 2016 I began writing a comment on this paper by first posting a series of blog posts questioning the validity of the underlying data and their implementation of the survey. The comment went through several rounds of review, was submitted, reviewed, revised and rejected at Land Econ (due to concerns about the DMT reply), submitted, reviewed, revised and then withdrawn from Economics E-Journal, and submitted, reviewed and accepted for publication at Ecological Economics. The comment goes further than the blog posts by showing that the adding-up test, though flawed in implementation and another hypothesis test is more appropriate, is actually supported in some tests using parametric WTP estimators. 

Desvousges, Mathews and Train (Ecological Economics, forthcoming) have now replied to my comment by describing 12 mistakes (12!) that I made. I agree that I made one of the mistakes on their list. I conducted an adding-up test by examining whether the confidence intervals for two willingness to pay estimates (the whole vs the sum of the parts) overlap. It is well-known that confidence intervals can overlap and yet the t-statistic for the test will indicate that the difference in means is statistically different. The mistake that I made was not checking the t-statistic. This is an embarrassing mistake. The worst part is that I teach this to undergraduates in the business statistics course. I tell them not to make this mistake and I've made it in a published journal article. I'm very embarrassed. 

There are a variety of reasons, though not excuses, for this mistake which I will describe in another blog post. But today, let me point out another mistake that I made that concerns me almost as much as the t-statistic: I used the wrong confidence intervals. In Whitehead (2020) I used the confidence intervals from the Delta Method (a first-order Taylor Series expansion from the variance-covariance matrix) which are symmetric. It is well-known that the distribution of a ratio of parameters (such as WTP) is not necessarily symmetric. The asymmetry gets more severe when the parameter in the denominator is imprecisely estimated as in Desvousges, Mathews and Train (Land Economics, 2015). Another approach that is common is the Krinsky-Robb (KR) confidence intervals. These are based on a simulation from the variance-covariance matrix of the estimated parameters. In a forthcoming blog post I'll show that the KR confidence intervals are very wide. So wide that the WTP for the sum of the parts lies within the confidence interval for the WTP for the whole, supporting the conclusions of Whitehead (2020). I'm embarrassed that I made this mistake too. 

My biggest concern, other than my big mistake (and the Delta Method confidence interval mistake) with the Desvousges, Mathews and Train "Reply to Whitehead" is that they do not take the problems with their own research very seriously. In contrast, when I've had papers that have received comments I've tried to learn from the comment and then tried to fix my paper (e.g., see Whitehead, Land Econ, 2004). Desvousges, Mathews and Train instead adopt the strategy that the best defense is a good offense. Their attitude seems to be that their data is no worse than any other CVM data set (in particular, they point to my own data from 15-30 years ago in footnote 3). I don't believe that this approach is the best way to advance economic science. 

My comment on Desvousges, Mathews and Train (Land Economics, 2015) addresses three main issues: (1) the data are flawed/low quality, (2) implementation of the adding-up test in the survey is flawed and (3) additional statistical tests for adding-up do not support the DMT (2015) results. None of these issues are refuted by Desvousges, Mathews and Train (forthcoming). Instead, each of these issues has been confused by the Desvousges, Mathews and Train "Reply to Whitehead". 

First, I provide a correction to my mistaken describe above. Second, here is a response to my 12 "mistakes":

(1) The log-linear models are not meaningless as claimed by DTM. The log-linear model and median WTP is a simple way of addressing negative WTP. The fact that the mean WTP from these models is infinite is not a functional form problem, it is a data problem. The median of the sum of WTP estimates provided by DMT (2020) lies within the 95% Krinsky-Robb confidence interval for the median WTP of the whole scenario. [more here]

(2) The linear-in-bid model that allows negative willingness to pay is not inappropriate. Negative WTP can arise from this functional form if the percentage of yes responses is less than 50% at the lowest bid or if the WTP estimate is statistically imprecise. The point estimate of mean WTP for this model provides positive WTP estimates in four out of the five scenarios. The negative WTP estimate is from the troublesome second scenario data (see (4)). The third scenario generates negative WTP values from the statistical distribution. Accounting for these in a statistical adding-up test supports my result that the sum of the WTP parts can not be statistically distinguished from the whole scenario. [more here]

(3) Following the approach taken in the correction, the adding-up test passes when respondents with missing demographics are dropped when the more appropriate confidence intervals (KR) are used. [more here]

(4) The weighted data does not support the results in Desvousges, Mathews and Train (Land Economics, 2015) as DMT claim. The weighted data with the whole and second scenarios are "roller coaster" and "Nike swoosh+ shaped instead of downward sloping as required by theory. This suggests that the weighted data reveals some irrationality amongst respondents. DMT's approach is to impose respondent rationality across the scenarios. They constrain the cost coefficients to be equal across scenarios in order to impose a downward sloping cost effect. This is inappropriate when it is done to hide statistically insignificant (roller coaster) and wrong-signed (Nike swoosh) slope coefficient. [more here]

(5) DMT notice that I conducted an adding-up test with the Kristrom nonparametric estimator in a 2016 blog post (here). They claim that I "inadvertently dropped observations" when conducting these calculations. Dropping these observations was not "inadvertant." In the blog post at issue I used a sample size of n=950 which is the same sample size that DMT (2015) used in their Table 5 (dropping observations with a missing age variable).

DMT (2020) report that the adding-up test fails with the Kristrom estimator and I "failed to report relevant findings" because I did not include this in Whitehead (2020). This begs the question: how many additional tests should be conducted in a comment on a paper? In Whitehead (2020) I provided three parametric tests using some the standard models in the literature. I then consider the robustness of these tests with (a) weighted data and (b) the complete case data set (n=934 after dropping those with missing age and income). 

(6) Claims that the Chapman et al. (2009) data and a number of my own data sets (circa 1992 - 2011) are of the same low quality as the Desvousges, Mathews and Train (Land Economics, 2015) data are overstated. I showed in an Appendix in Whitehead (2020) that the Chapman et al. (2009) are far superior in quality to the Desvousges, Mathews and Train (Land Economics, 2015) data. Using the length of the upper tail as a measure of quality, I find that my own data mostly ranges between the Chapman and DMT data (one of my data sets is a literal "off the chart" low quality outlier). Quality is an increasing function of sample size. [more here]

(7) Desvousges, Mathews and Train (2015) have not provided their internet survey for review. I asked twice. The first time Bill Desvousges had his assistant send me the Chapman et al. (2009) report containing their in-person surveys. The second time I asked he notified the Economics: E-Journal editor about my request. The editor told me that he thought I had everything I needed to write my replication paper and not to email Bill Desvousges again (I won't). Claims that their survey conveys information about substitution effects to survey respondents are simply assertions. It would be forthright to provide the survey for review.  

(8) DMT (2020) are correct by pointing out that "implicit claim" may be poor word choice. In DMT (2015) they have an empirical finding that there are no income effects. But, in mistake (9) they acknowledge that there is a statistically significant income coefficient when they use the weighted data. They have not explained why they chose to impose this "external" income constraint instead of incorporating income effects "internally" in the survey scenarios. Internal/external may be better word choice than implicit/explicit. 

(9) My statistically significant income coefficient was found using the models with weighted data. Desvousges, Mathews and Train (2020) state that they re-ran their simulations with the weighted income coefficient and found similar results. But, if they re-ran their simulations with the weighted income coefficient they should have done the test with the weighted WTP models, which lack validity (see (4) above). The "external" income test can not be conducted in a model with consistent assumptions made about the data unless one constrains the cost coefficients to be equal (which is done to hide statistically insignificant and wrong-size cost coefficients). 

In Whitehead (2020) also doubt that income is the correct budget constraint. I suspect that survey respondents have some environmental contribution budget in mind when answering CVM questions. In footnote 4 DMT state that this is a violation of microeconomic theory. I assume that they are referring to neoclassical microeconomic and ignore behavioral economics. Even then, a two-stage budgeting decision is consistent with two-stage budgeting where a household first allocates income to different budget categories and then maximizes subutility functions subject to the budget constraint (Deaton and Muellbauer 1980 -- this theory led to the development of the Almost Ideal Demand System econometric model). 

(10) My proposed hypothesis, based on my read of Desvousges, Mathews and Train (Land Economics, 2015) and the lack of the survey instrument (see (7)), is a one-tailed scope test. It is not a one-tailed adding-up test. Note also, that any information provided in a CVM survey about "substitute" environmental goods can be interpreted by respondents as complements (see Whitehead and Blomquist, WRR, 1991).

(11) Arrow et al. (1994) regret using the term adequate (in reference to the size of scope effects) in the NOAA Panel report. Instead they suggested the appropriate word is plausible scope effects. I pointed this out in Whitehead (Ecol. Econ 2016) and proposed scope elasticity as a measure. Scope elasticity is a more useful measure of plausibility than the adding-up test is for adequacy given difficulties in conducting an adding-up test. 

(12) My Turnbull standard error estimates differ from DMT's (2015) standard errors. I applied the formulas in Haab and McConnell (2002) with pooled (smoothed) data. DMT (2020) report that they used the raw data to construct confidence intervals with the smoothed data WTP estimate. My estimates of the standard errors are larger than DMT's (2020). But, it seems like standard errors with the raw data (not smoothed) should be larger than standard errors from the smoothed data. DMT (2020) do not provide much information on this estimation so it is difficult to say more. 

Comments welcome!