Environmental Economics

"Exxon pushes back at #exxonknew study"

From TheHill's Overnight Energy email (March 1):

Exxon pushed back in a report released Thursday against claims from a study last year on its public statements on climate change.

The August study from two Harvard University -- one of whom, Naomi Oreskes, is an outspoken Exxon critic -- used content analysis on decades of Exxon advertisements and statements to conclude that it misled the public about climate change.

But Exxon retained Kimberley Neuendorf, a Cleveland State University content analysis expert, to rebut the conclusions.

"I have concluded that [the authors'] content analysis does not support the study's conclusions because of a variety of fundamental errors in their analysis. [The authors' content analysis lacks reliability, validity, objectivity, generalizability, and replicability," she said.

"These basic standards of scientific inquiry are vital for a proper content analysis, but they are not satisfied by the [authors'] study."

Neuendorf wrote the book that laid out the methods that the Harvard researchers cited.

In her report for Exxon, Neuendorf argues that the study suffered problems with sampling, coding, consensus measurement, lack of disclosure and other issues.

Read the study here.

This sounds too familiar.

Let's say you have coded your data and estimated your first model and, gulp, there are no results*; but then it turns out this is a @SurveyMonkey problem ...

Suppose you have a 3 x 3 experimental design in a stated preference survey with cost $100, $250, and $500 and benefit is 25%, 50% and 75% improvement (in quality or quantity) variables. You ask more than one choice question. SurveyMonkey spits out the data with a number code which you assume is ordinal. For example:

Cost	Code
$100	1
$250	2
$500	3

But actually you find that the code is not in order. For example:  

Cost	Code
$100	3
$250	2
$500	1

Similar things happen to the benefit variable. There is no pattern to the number code assignment across choice question. So, when coding the variable in your statistical software you must compare the the qualitative data (don't forget your labels John Whitehead!) to the quantitative data. 

SurveyMonkey recently "upgraded" their product and this quirk seems to be an unintended result. 

*For example, no scope effect! BP-funded economists are going to hop on that train big-time.

The cost of offshore drilling safety regulations seem low relative to the costs

In case you missed it over the holiday break:

The Trump administration is poised to roll back offshore drilling safety regulations that were put in place after the 2010 Deepwater Horizon oil rig disaster in the Gulf of Mexico that killed 11 people and caused the worst oil spill in American history.

A proposal by the Interior Department’s Bureau of Safety and Environmental Enforcement, which was established after the spill and regulates offshore oil and gas drilling, calls for reversing the Obama-era regulations as part of President Trump’s efforts to ease restrictions on fossil fuel companies and generate more domestic energy production.

Doing so, the agency asserted, will reduce “unnecessary burdens” on the energy industry and save the industry $228 million over 10 years. ...

Industry groups like the American Petroleum Institute have long opposed the safety rules, calling them “flawed and costly.” The fossil fuel interest group warned in 2015 that the regulations would reduce capital investment in the Gulf by $4 billion a year and threaten 50,000 industry jobs. ...

The Obama-era rules, written in 2016, tightened controls on blowout preventers, devices that are intended to stop explosions in undersea oil and gas wells, and called for rig operators to have third parties certify that the safety devices worked under extreme conditions. In the Deepwater Horizon spill, a supposedly fail-safe blowout preventer failed after a section of drill pipe buckled.

As always with industry estimates you need to put them in context (ignoring the need to recognize that industry cost estimates are usually exaggerated to avoid regulations). The U.S. Energy Information Administration says that Gulf of Mexico oil production is 1.6 million barrels per day (2016). If the price per barrel is $65, then that leads to revenues of about $37 billion annually. The cost of the Obama-era rules is $22.8 million per year, about 0.06% of revenues or $456 for each lost job. The regulatory costs seem low to me, especially considering that the BP Deepwater Horizon blowout lead to 11 deaths and damages of $17.2 billion. 

Note, that in contrast to API's warning that 50,000 industry jobs were at risk in 2015, the EIA said that "Oil production in federal Gulf of Mexico expected to continue increasing."

What am I missing?

A short comment on a discrete choice experiment in the AER*

A cool new paper is the first (?) to use a discrete choice experiment in the AER:

Mas, Alexandre, and Amanda Pallais. 2017. "Valuing Alternative Work Arrangements." American Economic Review, 107(12): 3722-59.

Abstract: We employ a discrete choice experiment in the employment process for a national call center to estimate the willingness to pay distribution for alternative work arrangements relative to traditional office positions. Most workers are not willing to pay for scheduling flexibility, though a tail of workers with high valuations allows for sizable compensating differentials. The average worker is willing to give up 20 percent of wages to avoid a schedule set by an employer on short notice, and 8 percent for the option to work from home. We also document that many job-seekers are inattentive, and we account for this in estimation.

I might be in too deep (ya think?) but there are some questionable statements made on the quality of stated preference data. Before I go on, let me emphasize that this is a cool paper. Stated preference research can push the frontier in many fields besides environmental economics. The problem is that environmental economic stated preference researchers must run the 110 high hurdles while this paper gets entered into the 100 dash (to use a track analogy). 

On page 3724:

In this paper we report estimates of worker valuations over alternative work arrangements from a field experiment with national scope. The experiment elicits preferences on work arrangements by building a simple discrete choice experiment into the application process for a national call center. In this way we employ a method that can flexibly back out a willingness to pay (WTP) distribution from close to real market transactions.⁹

I don't consider something a "field experiment" unless there is a real monetary transaction.

Here is footnote 9: 

Discrete choice experiments are an extension of the contingent valuation literature whereby rather than directly asking people for valuations over an attribute (the stated preference method), people are given the choice of two or more scenarios and are asked to choose their preferred option. These scenarios usually vary the attributes and the prices and WTP can be estimated using random utility models (McFadden 1973; Manski 1977). Choice experiments have been shown to have better properties relative to stated preference valuation methods (Hanley, Wright, and Adamowicz 1998). A question is whether these experiments, which are usually survey-based, correspond to actual market behavior. This is something we can overcome by embedding the choice in a real market setting. Diamond and Hausman (1994), who critique stated preference valuation methods, hypothesize that the problem with the approach is not methodological but due to “an absence of preferences” over the attributes they are being asked to value. This is far less of a concern here since we are asking people to make choices over realistic work arrangements.

Is this the sort of lazy assertion that flies in labor economics papers at the AER? It wouldn't fly at JEEM, JAERE, Land Econ, Env and Res Econ, Res and Energy Econ, MRE, and etc (basically any environmental and resource economics journal). 

So:

• "Choice experiments have been shown to have better properties relative to stated preference methods" Huh? See "CVM surveys suck so we're using choice experiments, Q.E.D. (#174)".
• A "discrete choice experiment" is one type of "stated preference method" and choice experiments have lots of problems just like contingent valuation. See "What is Contingent Valuation?".
• Market experience does not eliminate hypothetical bias. See Morgan et al. (2016) for the case of oyster demand. The same problem can be found in the context of trip taking decisions and voting. 

Here is one of the hypothetical situations:

This question might be inconsequential (i.e., your choice will not affect anything whatsoever). If so, there is little reason for respondents to take it very seriously. See Groothuis et al. (2017).

The authors collect more data in online survey. Here is one scenario (parenthetical added):

Imagine that you are applying for a new job in your [current line of work, same line of work as your last job], and you have been offered two positions. Both positions are the same as your [current/last] job in all ways, and to each other, other than the work schedule and how much they pay. Please read the descriptions of the positions below. 

Position 1) This position is 40 hours per week. The work schedule is Monday–Friday 9 am–5 pm. This position pays the same as your [current/last] job.

Position 2) This position is 40 hours per week. The work schedule in this position varies from week to week. You will be given your work schedule one week in advance by your employer. The hours can be morning through evening, weekdays and weekends, but not nights. This position pays “X” (e.g., "15% more than") your [current/last] job.

Which position would you choose?

Another hypothetical question for those having a "flexible job" is:

Suppose your primary employer gives you the option of working a fixed work schedule, Monday-Friday during the daytime. Under this arrangement you would continue to work your usual number of hours but once your schedule is set you may not change the times and days of work. In exchange for having this fixed rather than flexible schedule you would get [2/5/10/20/35]% higher pay. Would you agree to this arrangement if given the choice?

Labor markets are characterized by negotiation. It is difficult to imagine that every survey respondent did not see these questions as the opening round of a negotiation. In both questions, for example, the respondent might be saying I choose position 2 but will only take the job if I can negotiate higher pay. Market researchers face the same issues in a market setting. If you are asked to purchase a new and improved product at a higher price you might answer no, thinking that the product might be offered anything and your signal will result in a lower price. 

These are issues that stated preference folks get hassled about over and over again. In contrast, Ivy leaguers write a dismissive footnote and get an AER**. Ok, the whine is over. 

 

---

*I've been enjoying my semester long sabbatical from blogging (and teaching and dressing like a grown-up) ... until this. 

**And they compare their "field experiment" surveys to the online survey results to deal with a referee who thinks that Diamond's adding up test is a paper killer. Here is footnote 43:

Diamond (1996) recommends testing for internal consistency in contingent valuation surveys. We go further in Section IV by comparing WTP estimates in the market setting to estimates from a nationally representative survey.

Do you think BP is going to let you get past the adding up test by asking some more hypothetical questions? I think not!

What is contingent valuation?

Hey fisheries/econ people, is contingent valuation really on the way out? What's replacing it?

— Michael C. Russell (@ProfRussellCIE) October 20, 2017

I'd be interested how @johnwhitehead81 and @tim_env_econ would answer this question.

— Ngaio Hotte (@nhotte) October 21, 2017

I went on a rant (the kids call it a "tweet storm") and since I don't know how to thread my tweets (and I don't want to lose them when Congress shuts down twitter) I thought I'd post them here (with spell check turned on). Tim was also in on it. This happened after I presented a paper using "discrete choice experiment" (DCE) data in Tim's department (that was fun!) so the thing was on our minds.

Here we go:

• John: Yes, CVM is out. And “discrete choice experiments” are in. And, they are the same thing (under certain conditions)!
• John: They are the same thing when the chioce is a for/against referendum with varying cost and benefit of a movement away from the status quo.
• Tim: CV is a DCE with variation in one attribute (typically) and the price. DCE is more flexible but subject to potentially more biases.
• John: I’m not sure why folks are describing CVM as something other than DCE, other than to try to avoid BP funded criticism.
• Tim: To understand the criticism, follow the $.
• John: Given all the CVM nonsense, you’re kinda stupid to call your study CVM and not DCE (call me stupid)
• Tim: For fun, let’s write a paper called “Scope Effects in DCE” and just use one attribute and price and see if anyone notices.
• John: First, write the CVM paper and send to Land Econ (or Env Res Econ) and get rejected because of Hausmann, Diamond and McFadden 1/n
• Tim: Then global replace “scope” with “attribute” and “CVM” with “DCE” and get R&R at Env Res Econ (or Land Econ) 2/n
• John: I honestly believe that’s how dumb it’s become 3/n
• John: I’m working on an R&R where a referee says you can’t (i.e., it’s not allowed) to ask a follow up CVM question anymore 4/n
• John: We explicitly test for changes/biases with the second question 5/n
• John: Research is about trying new things and discovery, not following the rules of the authority figures 6/n
• John: The NRDA context is very different than RESEARCH, the consultants and those who read them uncritically don’t care to differentiate 7/7

Here is the brief off-the-top-of-my-head history of thought:

• Mitchell and Carson (Using Surveys to Value Public Goods, RFF Press, 1989) define CVM as a hypothetical yes/no response to a randomly assigned cost variable. Variations in attributes of the policy/program (aka "scope") is one internal validity test.
• Carson and Louviere (Env and Res Econ, 2011) define referendum CVM with a scope test as a special case of DCE (as in the John/Tim tweets above). Payment card and open ended elicitation questions are CVM only. 
• Carson and Czajkowski (chapter 9 in Hess and Daly 2014) make the point in Carson and Louviere a bit more clear
• Zawojska and Czajkowski (J Env Econ Policy 2017) take it one step too far and say that referendum CVM with a scope test is DCE and not CVM.

Who Knows What Willingness to Pay Lurks in the Hearts of Men?*

From the inbox:

The new issue of Econ Journal Watch is online at http://econjwatch.org.

In this issue:

Who Knows What Willingness to Pay Lurks in the Hearts of Men? John Whitehead argues that certain authors claim to know, based on their survey data, more than they do about people’s willingness to pay for a wetlands project. The article continues an exchange from the Journal of Environmental Economics and Management.

Here is the abstract:

The present article is the fourth piece in a sequence of articles, the three previous having appeared in Journal of Environmental Economics and Management. In JEEM, Kevin Egan, Jay Corrigan, and Daryl Dwyer published an article with findings from their survey of Ohio residents, collecting contingent valuation data for a wetlands project. They elicit willingness to pay (WTP) with annual and one-time payment schedules in dichotomous choice contingent valuation method questions and compare these to consumer surplus estimates derived from the travel cost method. Using a conservative nonparametric estimator, they conclude that WTP from the annual payment schedule is a “better match” with consumer surplus estimates. Combined with two other analyses, Egan, Corrigan, and Dwyer conclude that “these three arguments make a strong case for matching the duration of a [contingent valuation] survey’s payments with the duration of the proposed benefits, which will most often mean presenting survey respondents with ongoing annual payments” (2015, 134). Dissatisfied with their strong claims, I showed that the contingent valuation method data is of relatively low quality and, with an alternative nonparametric estimator, showed that WTP from the one-time payment schedule is a better match with the consumer surplus estimates. Given the low-quality data, the range of WTP estimates is too wide to draw any conclusions about the most appropriate payment schedule (Whitehead 2017a). In a reply, the authors defend the theoretical validity of their data by appealing to the statistical significance of the entire bid schedule. They conduct three sensitivity analyses, and write that they “overwhelmingly support our original conclusion in favor of using annual payments in” contingent valuation surveys (Egan, Corrigan, and Dwyer 2017). Here, I use parametric approaches to measure WTP and conduct a more complete sensitivity analysis. Egan, Corrigan, and Dwyer’s conclusion that the annual payment results “better match” the consumer surplus estimates is not supported by my analysis. This paper reinforces my initial contention that their contingent valuation data is not useful for the payment schedule issue.

Here is the article: Whitehead, John C., Who Knows What Willingness to Pay Lurks in the Hearts of Men? A Rejoinder to Egan, Corrigan, and Dwyer, Econ Journal Watch 14(3):346–361, September 2017. https://econjwatch.org/file_download/1036/WhiteheadSept2017.pdf?mimetype=pdf

*The Shadow knows ... a paper title suggested by EJW.

 

Comments on “Economic Analysis for the Proposed Definition of ‘Waters of the United States’—Recodification of Pre-Existing Rules”

Here are the first 622 of 5230 words:

Two years ago, the U.S. Environmental Protection Agency and U.S. Department of the Army (EPA-Army 2015) estimated that the total quantified benefits of the Clean Water Rule (CWR) range from a low of $339 million to $350 million (2014 dollars). The largest component of these benefits was due to wetland mitigation, which was estimated at $306 million (using the “original number of ORM2 other waters records” scenario). EPA-Army (2015) concluded that the benefits of the CWR exceed the costs.

The EPA and Army (2017) now state that the “largest and most uncertain estimates from the[ir] 2015 CWR … [economic analysis] are associated with the benefits of the CWA 404 program.” The agencies argue that the wetland mitigation benefits are uncertain for several reasons. In response to this uncertainty the agencies choose to dismiss the monetized benefits presented in their 2015 analysis, instead presenting them as qualitative benefits in EPA-Army (2017). They justify this decision by stating that they “believe the cumulative uncertainty in this context is too large to include quantitative estimates in the main analysis for this proposed rule” (page 9). In the low (high) end scenario (Tables 1 and 2 on page 10 in EPA-Army (2017)) the $314 million (2016 dollars) in wetland mitigation benefits from EPA-Army (2015) are replaced by “$B”.

Any conclusion that the benefits of the CWR are less than the costs is not supported by the current analysis. The EPA’s conclusion that the benefits are less than the cost is baseless.

There is no need for the agencies to resort to qualitative analysis of the wetland mitigation benefits. A review of the pre-2000 studies now dismissed by the agencies indicates that many of the concerns being raised by EPA-Army (2017) are not justified. Many of the studies used in EPA-Army (2015) conducted a scope test and other validity tests. Lessons from the temporal reliability literature suggest that environmental values produced in those types of studies are fairly stable over long periods of time. Further, meta-analyses of the broader wetland valuation literature suggest that there are systematic (i.e., explainable) differences in values over studies and over time.  That is, studies conducted in the past can be adjusted to account for methodological advances using insights from meta-analyses of the valuation literature, resulting in updated valid results.

In addition, there have been more recent wetland valuation studies that have appeared in the literature. We have identified several additional valuation studies published since 2000. These studies suggest that the wetland mitigation benefits estimated by the agencies in their original economic analysis (EPA-Army 2015) were accurately measured.

Finally, the agencies have taken two contrasting positions with regard to the uncertainty of wetland mitigation benefits. In EPA-Army (2015), the agencies used only the point estimate without sensitivity analysis around this estimate, ignoring the uncertainty. In EPA-Army (2017), the agencies again choose to avoid sensitivity analysis but this time they move to an extreme and avoid quantitative measurement of the benefits. Using sensitivity analysis as an appropriate strategy for addressing uncertainty is discussed below.

Considering these issues, the agencies’ decision to consider only qualitative wetland mitigation benefits appears to be an illogical overreaction to a normal level of uncertainty in the conduct of standard benefit-cost analysis of environmental policy. Even if the agencies now feel justified in only presenting qualitative wetland mitigation benefits, there is no evidence that these benefits would be so small as to reverse the sign on the net benefits calculation (i.e., positive net cost savings on page 20 in EPA-Army (2017)). The only way to justify a negative net benefit calculation would be to conduct the appropriate sensitivity analysis, as discussed below, and show that negative net benefits are more likely than positive net benefits, which was not done.

Here is:

• Some background: http://www.env-econ.net/2017/08/when-you-want-environmental-benefits-to-be-zero-.html
• My full report conducted for the Southern Environmental Law Center (SELC):  Download 2017-09-27 Whitehead Comments
• The SELC's letter:  Download 2017-09-27 SELC CWR Comments EPA-HQ-2017-203
• The 2017 EPA-Army benefit-cost analysis: Download 2017 Economic Analysis Repeal Rule
• The 2015 EPA-Army benefit-cost analysis: Download 2015-05-20 Economic Analysis CWR

See also: 

• Kevin J. Boyle, Matthew J. Kotchen, V. Kerry Smith, Deciphering dueling analyses of clean water regulations, Science 06 Oct 2017: Vol. 358, Issue 6359, pp. 49-50 [View Full Text]; Summary: Government agencies are often required to conduct benefit-cost analyses for major regulatory actions (1). When benefit-cost analysis is consistent with best practices, it provides a systematic and science-based approach for informing policy and regulatory decisions. It has been particularly important for health and environmental regulations. Yet the wide disparity between the quantified benefits in two recent and conflicting regulatory impact analyses (RIAs) related to the U.S. Clean Water Act (CWA) has the potential to undermine the credibility of agencies' benefit-cost analyses. It also highlights the need for a more systematic protocol that ensures the information base is adequate and appropriately applied to support agency analyses and public decision-making. This includes applications in the context of the CWA, which is the focus of an 11 October hearing in the U.S. Supreme Court.
• Jason Schwartz and Jeffrey Shrader, Muddying the Waters: How the Trump administration is obscuring the value of wetlands protection from the Clean Water Rule, Institute for Policy Integrity, September 2017 http://policyintegrity.org/files/publications/Muddying_the_Waters.pdf
  
  

A replication of willingness-to-pay estimates in ‘An adding up test on contingent valuations of river and lake quality’ (Land Economics, 2015)

This has been posted as a discussion paper at the Economics E-journal. Here is the abstract and citation:

Desvousges, Mathews and Train (2015) find that their contingent valuation method (CVM) survey data does not pass the adding up test using a nonparametric estimate of mean willingnessto-pay. Their data suffers from non-monotocity, flat bid curve and fat tails problems, each of which can cause willingness-to-pay estimates to be sensitive to the approach chosen to measure the central tendency. Using additional parametric approaches that are standard in the literature, I find that willingness to pay for the whole is not statistically different from the sum of the parts in two of three additional estimates. In additional robustness checks, all six of the additional tests find that the WTP estimates do not reject the adding up hypothesis. The negative result in Desvousges, Mathews and Train (2015) is not robust to these alternative approaches to willingness-to-pay estimation.

John C. Whitehead (2017). A replication of willingness-to-pay estimates in ‘An adding up test on contingent valuations of river and lake quality’ (Land Economics, 2015). Economics Discussion Papers, No 2017-55, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2017-55

The Economics E-Journal:

... is an online-only journal dedicated to publishing high quality original research across all areas of economics. It views academic publishing as a cooperative enterprise between authors, editors, referees, and readers.

The paper and the authors' reply has been posted in the Replication Section:

Replications are an important public good to the economics community, and as such, they tend to be under-valued. By opening a replications section in our open access, open assessment journal, we hope to make it easier for authors to provide this important public good.

The peer review process is open. My paper has passed through the first part of the first stage of the review process:

The author submits a paper, if possible with hyperlinks to the referenced literature. In addition to the paper, the author is encouraged to also provide mathematical derivations, underlying data sets and a list of researchers who may be interested in reviewing and rating the paper.

The assistant editors check formal and professional standards and pass the paper to a co-editor in the relevant research field.

The co-editor decides on whether to accept the paper for the further peer review process by answering two questions: (i) Whether the paper shows promise of making a significant contribution and (ii) whether it meets basic scientific standards. The co-editor is free to ask other experts in the field for a quick opinion.

If these questions are answered in the affirmative, the paper is promptly published on the platform of the Economics discussion Papers series. ...

In other words, I didn't get a "desk reject." The second part of the first stage of the review process is different from a regular journal in that it is open and anyone can participate (not just two to three plus anonymous referees like a regular journal):

The co-editor appoints at least two referees for reviewing the paper .... The referee reports are uploaded on the discussion paper platform within six weeks if possible. The referees’ comments may be anonymous, but referees are encouraged to allow their reports to be attributed. Referees are asked to provide short reports that focus on two questions: (i) Is the contribution of the paper potentially significant? (ii) Is the analysis correct? Referees are asked NOT to rewrite the paper or to change its focus. With their assignment thus circumscribed, the referees can be expected to provide reports before the six week deadline elapses.

While the referees are reviewing the paper, the corresponding public discussion platform is open for eight weeks (the "discussion period"). On this platform, registered readers (and invited readers) may review the paper by uploading comments (anonymously or with the name of the contributor published on the Web). In addition, the referees´ comments are posted on the Web site. Authors are asked to respond to these comments. All comments and responses are published and archived in the Economics Discussion Paper section. ...

During the discussion period, the editorial staff sends the paper to potentially interested researchers (invited readers) to encourage them to also post a comment on the discussion platform. The list of these researchers is compiled partly by the managing staff with the help of the Leibniz Information Centre for Economics and partly by the author. The author is free to respond to all comments.

The second stage of the review process begins with the "reject" vs "revised and resubmit" decision:

Based on the referee reports, the comments made by registered and invited readers and the author replies, the co-editor decides whether the paper is accepted or rejected for publication in Economics.

If the co-editor´s decision deviates from the views of the referees and the registered readers, the co-editor is free to justify it publicly through a statement in the paper’s discussion forum. In the case of acceptance, the co-editor prepares a short statement, e.g. containing suggestion for revisions.

If accepted, the discussion paper or its revised version is published as article in Economics. The discussion paper with all comments is permanently archived and remains accessible to the public for documenting the paper’s history.

I anticipate that this will be a fascinating process and I'm sure to learn a lot. 

"How does a Nobel-prize-winning economist become a victim of bog-standard selection bias?"

Andrew Gelman:

Again, I’m not saying that Heckman and his colleagues are doing this. I can only assume they’re reporting what, to them, are their best estimates. Unfortunately these methods are biased. But a lot of people with classical statistics and econometrics training don’t realize this: they thing regression coefficients are unbiased estimates, but nobody ever told them that the biases can be huge when there is selection for statistical significance.

And, remember, selection for statistical significance is not just about the “file drawer” and it’s not just about “p-hacking.” It’s about researcher degrees of freedom and forking paths that researchers themselves don’t always realize until they try to replicate their own studies. I don’t think Heckman and his colleagues have dozens of unpublished papers hiding in their file drawers, and I don’t think they’re running their data through dozens of specifications until they find statistical significance. So it’s not the file drawer and it’s not p-hacking as is often understood. But these researchers do have nearly unlimited degrees of freedom in their data coding and analysis, they do interpret “non-significant” differences as null and “significant” differences at face value, they have forking paths all over the place, and their estimates of magnitudes of effects are biased in the positive direction. It’s kinda funny but also kinda sad, that there’s so much concern for rigor in the design of these studies and in the statistical estimators used in the analysis, but lots of messiness in between, lots of motivation on the part of the researchers to find success after success after success, and lots of motivation for scholarly journals and the news media to publicize the results uncritically. These motivations are not universal—there’s clearly a role in the ecosystem for critics within academia, the news media, and in the policy community—but I think there are enough incentives for success within Heckman’s world to keep him and his colleagues from seeing what’s going wrong.

Again, it’s not easy—it took the field of social psychology about a decade to get a handle on the problem, and some are still struggling. So I’m not slamming Heckman and his colleagues. I think they can and will do better. It’s just interesting, when considering the mistakes that accomplished people make, to ask, How did this happen?

via andrewgelman.com

Thinking about this in my small world, searching for that one willingness to pay (WTP) estimate that delivers statistical significance that favors your hypothesis is the equivalent of p-hacking. The state-of-the-art, NOAA-Panel-endorsed referendum vote question can be quite sensitive to the nonparametric or parametric estimator chosen to summarize the votes into WTP. This is especially true when the data are "difficult" (Haab and McConnell). When a WTP estimate varies by factors of 3 or 4 depending on whether a nonparametric or parametric WTP estimate is employed then researchers should interpret significance tests conducted with these data very carefully (if not reject them outright). I've become very worried that there is "lots of motivation for scholarly journals ... to publicize the results uncritically." Researchers who have "difficult" CVM data need to present the full range of WTP estimates.  

Field research: Here I am doing what it takes* to effectively field a stated preference survey

*Doing what it takes: "Hiking" the 4.8 mile roundtrip Grandfather Trail (the last half mile featured thunder, lightning, pouring rain and some hail) at Grandfather Mountain State Park to better understand Hemlock Woolly Adelgid (I think that dead tree in the middle is a Hemlock). The last time I hiked this trail was 7 years ago (enough time to forget how difficult it is) and resulted in my rugged profile picture. Here is another picture from the hike (about 20 minutes before the storm began):