Environmental Economics

What is contingent valuation?

Hey fisheries/econ people, is contingent valuation really on the way out? What's replacing it?

— Michael C. Russell (@ProfRussellCIE) October 20, 2017

I'd be interested how @johnwhitehead81 and @tim_env_econ would answer this question.

— Ngaio Hotte (@nhotte) October 21, 2017

I went on a rant (the kids call it a "tweet storm") and since I don't know how to thread my tweets (and I don't want to lose them when Congress shuts down twitter) I thought I'd post them here (with spell check turned on). Tim was also in on it. This happened after I presented a paper using "discrete choice experiment" (DCE) data in Tim's department (that was fun!) so the thing was on our minds.

Here we go:

• John: Yes, CVM is out. And “discrete choice experiments” are in. And, they are the same thing (under certain conditions)!
• John: They are the same thing when the chioce is a for/against referendum with varying cost and benefit of a movement away from the status quo.
• Tim: CV is a DCE with variation in one attribute (typically) and the price. DCE is more flexible but subject to potentially more biases.
• John: I’m not sure why folks are describing CVM as something other than DCE, other than to try to avoid BP funded criticism.
• Tim: To understand the criticism, follow the $.
• John: Given all the CVM nonsense, you’re kinda stupid to call your study CVM and not DCE (call me stupid)
• Tim: For fun, let’s write a paper called “Scope Effects in DCE” and just use one attribute and price and see if anyone notices.
• John: First, write the CVM paper and send to Land Econ (or Env Res Econ) and get rejected because of Hausmann, Diamond and McFadden 1/n
• Tim: Then global replace “scope” with “attribute” and “CVM” with “DCE” and get R&R at Env Res Econ (or Land Econ) 2/n
• John: I honestly believe that’s how dumb it’s become 3/n
• John: I’m working on an R&R where a referee says you can’t (i.e., it’s not allowed) to ask a follow up CVM question anymore 4/n
• John: We explicitly test for changes/biases with the second question 5/n
• John: Research is about trying new things and discovery, not following the rules of the authority figures 6/n
• John: The NRDA context is very different than RESEARCH, the consultants and those who read them uncritically don’t care to differentiate 7/7

Here is the brief off-the-top-of-my-head history of thought:

• Mitchell and Carson (Using Surveys to Value Public Goods, RFF Press, 1989) define CVM as a hypothetical yes/no response to a randomly assigned cost variable. Variations in attributes of the policy/program (aka "scope") is one internal validity test.
• Carson and Louviere (Env and Res Econ, 2011) define referendum CVM with a scope test as a special case of DCE (as in the John/Tim tweets above). Payment card and open ended elicitation questions are CVM only. 
• Carson and Czajkowski (chapter 9 in Hess and Daly 2014) make the point in Carson and Louviere a bit more clear
• Zawojska and Czajkowski (J Env Econ Policy 2017) take it one step too far and say that referendum CVM with a scope test is DCE and not CVM.

"How does a Nobel-prize-winning economist become a victim of bog-standard selection bias?"

Andrew Gelman:

Again, I’m not saying that Heckman and his colleagues are doing this. I can only assume they’re reporting what, to them, are their best estimates. Unfortunately these methods are biased. But a lot of people with classical statistics and econometrics training don’t realize this: they thing regression coefficients are unbiased estimates, but nobody ever told them that the biases can be huge when there is selection for statistical significance.

And, remember, selection for statistical significance is not just about the “file drawer” and it’s not just about “p-hacking.” It’s about researcher degrees of freedom and forking paths that researchers themselves don’t always realize until they try to replicate their own studies. I don’t think Heckman and his colleagues have dozens of unpublished papers hiding in their file drawers, and I don’t think they’re running their data through dozens of specifications until they find statistical significance. So it’s not the file drawer and it’s not p-hacking as is often understood. But these researchers do have nearly unlimited degrees of freedom in their data coding and analysis, they do interpret “non-significant” differences as null and “significant” differences at face value, they have forking paths all over the place, and their estimates of magnitudes of effects are biased in the positive direction. It’s kinda funny but also kinda sad, that there’s so much concern for rigor in the design of these studies and in the statistical estimators used in the analysis, but lots of messiness in between, lots of motivation on the part of the researchers to find success after success after success, and lots of motivation for scholarly journals and the news media to publicize the results uncritically. These motivations are not universal—there’s clearly a role in the ecosystem for critics within academia, the news media, and in the policy community—but I think there are enough incentives for success within Heckman’s world to keep him and his colleagues from seeing what’s going wrong.

Again, it’s not easy—it took the field of social psychology about a decade to get a handle on the problem, and some are still struggling. So I’m not slamming Heckman and his colleagues. I think they can and will do better. It’s just interesting, when considering the mistakes that accomplished people make, to ask, How did this happen?

via andrewgelman.com

Thinking about this in my small world, searching for that one willingness to pay (WTP) estimate that delivers statistical significance that favors your hypothesis is the equivalent of p-hacking. The state-of-the-art, NOAA-Panel-endorsed referendum vote question can be quite sensitive to the nonparametric or parametric estimator chosen to summarize the votes into WTP. This is especially true when the data are "difficult" (Haab and McConnell). When a WTP estimate varies by factors of 3 or 4 depending on whether a nonparametric or parametric WTP estimate is employed then researchers should interpret significance tests conducted with these data very carefully (if not reject them outright). I've become very worried that there is "lots of motivation for scholarly journals ... to publicize the results uncritically." Researchers who have "difficult" CVM data need to present the full range of WTP estimates.  

I'm cleaning out my email inbox (1800+ messages)

I'm sending this message to a bunch of people that sent me a message that I never replied to (and I should have):

Sorry that I never got back to you on this ... I just stepped down as department chair and I hope to be a better human being sometime in the near future.

My new autoresponse to appstate.edu emails

Hello,

John Whitehead is happy to report that he is no longer a part-time academic administrator (i.e., Economics Department Chair). If your email is regarding department business, please redirect it to our new chair Todd Cherry at cherrytl at appstate.edu.

John Whitehead is also happy to report that he is on Off-Campus Scholarly Assignment (i.e., sabbatical) in Fall 2017 and back to teaching, research and service in Spring 2018. If your email is about non-department business, he'll reply ASAP (or maybe not).

Thanks,

John Whitehead

P.S. For the remainder of his academic career, John Whitehead will refer to himself in the third person.

A history of the Exxon funded CVM debate makes it to twitter ... in case you are missing it, there is a more recent BP funded CVM debate @Undercoverhist

1/ Looooong tweetstorm on use of contingent valuation method in Exxon Valdez case & what it says on how economics science & expertise work pic.twitter.com/SohMYTwnh7

— Beatrice Cherrier (@Undercoverhist) June 25, 2017

These 48 tweets cover the article that Tim posted earlier and also this one by Spencer Banzhaf (who alerted us to the Maas and Svorcencik article): 

Banzhaf , H. Spencer, Constructing Markets: Environmental Economics and the Contingent Valuation Controversy (October 21, 2016). History of Political Economy, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2959427.

Having lived through it [see below], I thought I knew everything about the CVM debate but still learned something new. For example: Diamond, Hausman and McFadden did not collect any of the Exxon-funded data (i.e., write the surveys, focus group, pretest and field the surveys). It seems that it was collected by others [*.pdf] and then, given their prominence in the profession, Diamond, Hausman and McFadden did the econometrics. 

Tweets 4 and 5 say:

4/Rather than countering w/ alternative estimates, Maas&Svorencik explain, Exxon casted doubt on method itself: contingent valuation (CV)

5/Unlike tobacco/climate cases, Exxon did NOT manufacture doubt, but tapped into EXISTING debates on scientific character of CV among econs

Being economic history, neither of these articles mention the brand new BP funded critique of CVM. It is clear that BP is following the same game plan as Exxon. Here is my recommended readings on the CVM Debate - Part Deux:

1. Carson, Richard T. "Contingent valuation: A practical alternative when prices aren't available." The Journal of Economic Perspectives 26, no. 4 (2012): 27-42.
2. Hausman, Jerry. "Contingent valuation: from dubious to hopeless." The Journal of Economic Perspectives 26, no. 4 (2012): 43-56.
3. Kling, Catherine L., Daniel J. Phaneuf, and Jinhua Zhao. "From Exxon to BP: Has some number become better than no number?." The Journal of Economic Perspectives 26, no. 4 (2012): 3-26.
4. [*] Desvousges, William, Kristy Mathews, and Kenneth Train. "Adequate responsiveness to scope in contingent valuation." Ecological Economics 84 (2012): 121-128.
5. Haab, Timothy C., Matthew G. Interis, Daniel R. Petrolia, and John C. Whitehead. "From hopeless to curious? Thoughts on Hausman's “dubious to hopeless” critique of contingent valuation." Applied Economic Perspectives and Policy 35, no. 4 (2013): 593-612.
6. [*] Desvousges, William, Kristy Mathews, and Kenneth Train. "An adding-up test on contingent valuations of river and lake quality." Land Economics 91, no. 3 (2015): 556-571.
7. Dickinson, David L., and John C. Whitehead. "Dubious and dubiouser: Contingent valuation and the time of day." Economic Inquiry 53, no. 2 (2015): 1396-1400. [note: published in the Miscellany section]
8. [*] Desvousges, William, Kristy Mathews, and Kenneth Train. "From Curious to Pragmatically Curious: Comment on “From Hopeless to Curious? Thoughts on Hausman's ‘Dubious to Hopeless’ Critique of Contingent Valuation†”." Applied Economic Perspectives and Policy 38, no. 1 (2016): 174-182.
9. Haab, Timothy C., Matthew G. Interis, Daniel R. Petrolia, and John C. Whitehead. "Interesting Questions Worthy of Further Study: Our Reply to Desvousges, Mathews, and Train's (2015) Comment on Our Thoughts (2013) on Hausman's (2012) Update of Diamond and Hausman's (1994) Critique of Contingent Valuation." Applied Economic Perspectives and Policy 38, no. 1 (2016): 183-189.
10. Whitehead, John C. "Plausible responsiveness to scope in contingent valuation." Ecological Economics 128 (2016): 17-22. [note: this is a comment on DMT (2012) that grew into a full length paper with original content]
11. Chapman, David J., Richard C. Bishop, W. Michael Hanemann, Barbara J. Kanninen, Jon A. Krosnick, Edward R. Morey, and Roger Tourangeau. "On the adequacy of scope test results: Comments on Desvousges, Mathews, and Train." Ecological Economics 130 (2016): 356-360.
12. [*] Desvousges, William. "Reply to ‘On the adequacy of scope test results: Comments on Desvousges, Mathews, and Train’." Ecological Economics 130 (2016): 361-362.
13. Poe, Gregory L. "Behavioral Anomalies in Contingent Values and Actual Choices." Agricultural and Resource Economics Review 45, no. 2 (2016): 246-269.
14. Whitehead, John C. "A Comment on 'An Adding Up Test on Contingent Valuations of River and Lake Quality'”. Appalachian State University Department of Economics Working Paper No. 17-01. 2017.
15. Bishop, Richard C., Kevin J. Boyle, Richard T. Carson, David Chapman, W. Michael Hanemann, Barbara Kanninen, Raymond J. Kopp et al. "Putting a value on injuries to natural assets: The BP oil spill." Science 356, no. 6335 (2017): 253-254.
16. [*] McFadden, Daniel, and Kenneth Train, eds. Contingent Valuation of Environmental Goods: A Comprehensive Critique. Edward Elgar Publishing, 2017.
17. Whitehead, John C. "A Replication of Willingness-to-pay Estimates in ‘An Adding Up Test on Contingent Valuations of River and Lake Quality’ (Land Economics, 2015)." unpublished manuscript, June 2017.

BP funded research is indicted by the [*]. 

Other than this:

Whitehead, John C., "Natural Resource Damage Assessment … May be Hazardous to Your Health," AERE Newsletter 36(2):32-38, November 2016.  [Download Aere newsletter november 2016 finalmv revised -essay]

What have I missed on my reading list?

Another referee rant

From the inbox:

Ms. Ref. No.: [redacted]
Title: [article title] [journal]

This is an automated reminder. If you already submitted your review, or if you have sent us an email regarding your review, please disregard this reminder.

Dear Dr. Whitehead,

You kindly accepted our invitation and agreed to review the aforementioned manuscript on May 03, 2017. Your review of this manuscript (please see abstract below) is now overdue. The review was due on: May 23, 2017.

We would be grateful if you could submit your review of this manuscript as soon as possible.

If however you are unwilling or unable to review this particular manuscript, please reply to this email and let us know. We would be grateful for any recommendations for alternate reviewers you can give.

My reply:

[journal],

This review is next up. I had a few ahead of it and didn't realize that the deadline was only 3 weeks from the time I accepted the assignment.

Thanks,

John Whitehead

P.S. 3 weeks? Is there any academic who happens to deal with the normal craziness of work life that can drop everything and conduct and [sic] unpaid review in that amount of time? I'd like to see your stats on whether short deadlines (a) changes the acceptance rate and (b) increases the time to decision. As far as I'm concerned, I won't accept any more review assignments that so casually dismiss the value of unpaid referee time.

I suspect I might be part of a referee experiment. Group A is treated normally (4-6 week deadline) and Group B is treated like chumps (3 week deadline). The hypothesis is that if you treat referees like chumps then they will act more like chumps. Behavioral economics, I guess. 

 

Drink at each awkward pause / stupid mistake / embarrassing gesture

I'm a "research parasite" / "party pooper" (without apology)

But haven't won the award (or made many friends) yet (Congratulations, you're a parasite!):

How would you feel if you were called a parasite in a respected scientific journal? If you’re Drs. Erick Turner and Kun-Hsing Yu, the answer is: elated.

The two scientists have each received an inaugural Research Parasite award, for which they’re honored in the latest issue of Nature Genetics.

The tongue-in-cheek honor comes from the mind of computational biologist Iddo Friedberg, and was brought to life by Casey Greene, a pharmacologist at the University of Pennsylvania. The idea was a response to a now-infamous editorial in the New England Journal of Medicine that referred to those who picked over the bones of previously published studies as “research parasites.”

The award has the serious aim of fostering “the craft of data reanalysis for novel ends.” And the particular brand of parasitism that Turner and Yu demonstrated shows just how useful the rehashing of others’ findings can be. ...

Data sharing is not a new idea. Since 2003, the National Institutes of Health has required grantees who receive at least $500,000 per year in funding to share their results. And many journals demand that authors make their data public as a prerequisite of publication, although those polices are often not enforced.

“Research parasites help to maintain the self-correcting nature of science,” Greene and his colleagues wrote in Nature Genetics announcing the winners. “Scientists who perform rigorous parasitism put scientific work to the test, and their results may support or challenge what we think we know.”

This year’s contest generated 41 applications and the group is already accepting applications for 2018. Greene says he’s pleased with the results so far, but he fears that some scientists may be turned off by the satire. “I’d like to transition to the positive framing of the award — awards for rigorous secondary data analysis — and to work harder to encourage members of underrepresented groups to apply,” he said.

Turner said science does not value secondary data analysis as much as it should. “My cynical sense is that one’s academic ‘success’ is largely measured in terms of how much grant money you bring in — not only for your work but also for the institution via indirect costs,” he told STAT. ”Funding agencies seem enamored with chasing the next big thing, even though the ‘gold in them thar hills’ often turns out to be fool’s gold. The research parasite who reveals that fact may be viewed by many as a party pooper.”

Hat tip: Retraction Watch

I've written more comments than most over the years. My first publication was a comment. Much of my research time over the past 6 months has been dominated by this one. And there's been this and this and this and this. Not all are reanalyses of the authors' data but the same jerkish behavior is evident in each. It seems that if something doesn't seem quite right then you should try to point it out. 

Your daily reminder ...

... that I'm a very famous sports economist*:

This study shows that fans and people living in the region of 28 Football Bundesliga teams from all three divisions are willing to support their team financially. Survey respondents were asked for their willingness-to-pay to avoid a negative outcome (e.g., relegation) and to achieve a positive outcome (e.g., promotion). Fan bonds are applied as an alternative payment vehicle within the contingent valuation method. The results show that different factors affect the decision to support the team and the actual amount of willingness-to-pay—for attendees and nonattendees. Public goods are particularly relevant for reporting a positive willingness-to-pay. (JEL Z23, L83, H41)

Wicker, P., Whitehead, J. C., Johnson, B. K. and Mason, D. S. (2016), WILLINGNESS-TO-PAY FOR SPORTING SUCCESS OF FOOTBALL BUNDESLIGA TEAMS. Contemporary Economic Policy, 34: 446–462. doi: 10.1111/coep.12148

via onlinelibrary.wiley.com

One of my favorite quotes from the Western meetings is that "the CVM is like Wal Mart...." I forget the rest but it kind of says it all, right?

*And athlete, here I am draining a three in the High Country Senior Games 3v3 basketball on Wednesday night. The without-their-big-man High Country 50+ lost to the Ashe Panthers 55+ by a handful of points. Three of us fouled out (huh?) and we finished the game with only two guys on the court. You gotta have a big man in Senior Games. 

Burn!

From Referee #2:

Unfortunately, I find the exposition of the paper rather to be substantially weaker than I would expect for an article of this type at any journal .... I found several aspects of the process for how the survey questions were assigned or presented to respondents, and how the analysis was conducted and interpreted, to be either opaque or at least sloppy in explanation. I would normally consider it a minor comment, but I also found the failure to use commas in a myriad of numerous instances (redundancy for emphasis) to contribute to a frustrating read, as well as occasional failures to recognize that the noun associated with a verb in the sentence (especially a second verb, such as following the word “and”) could not be relaying the meaning that the authors intended.

Guess the rejected paper based on this comment and I'll buy you a beer.