I've spent most of the time since grades were submitted catching up on reviews. I'm getting old and tired of writing the same things over and over. For example, I tend to referee a few revealed and stated preference data combination papers each year. The data are typically collected from a survey eliciting recreational behavior. Data points should include:
- ex-post revealed preference (RP) trips
- ex-ante stated preference (SP) trips under most recent conditions (i.e., status quo)
- SP trips under different conditions
Sometimes #2 is left out. Other times #1 and #2 are assumed equivalent in the empirical model. Here is a comment that I typically must include in a referee report:
The model includes a revealed preference trip question and a set of stated preference questions. This raises the possibility that hypothetical bias is present in all of the scenario change questions. Hypothetical bias can affect demand intercepts and shifts. This potential for bias needs to be explicitly addressed. See Whitehead et al., Marine Resource Economics 23(2):119-135, 2008.
I first heard the flippant remark that ex-post revealed preference trip data and ex-ante stated preference status quo trip data are essentially equivalent at a Camp Resources in Wilmington a long time ago. This is simply an assertion that has been shown to not always be true. If #1, #2 and #3 data points are collected then SP dummy (the kids call these "fixed effects") and slope change variables should be included in the model to test for hypothetical bias (i.e., RP and SP demands may be different). If data point #2 is not collected then the paper simply can not be used for policy analysis unless explicit caveats are made. For example, here is the caveat from Hynes and Greene, Land Economics (2012):
Also, it should be noted that Whitehead et al. (2008a) have shown that when the product of trips and consumer surplus per trip is taken as an estimate of consumer surplus per year in contingent behavior models, hypothetical bias may lead to upwardly biased seasonal consumer surplus estimates.12 While this paper's contribution does not require the avoidance of this problem, we mention it in order to caution the reader who might wish to use the results for policy analysis.
--------------
12A possible solution to this problem is for the researcher to first ask respondents to report contingent behavior under the circumstances of no change in quality or site status prior to asking them to report contingent behavior under the circumstances of the change in the status quo.
I referee several other types of empirical paper that gives short shrift to data reporting. Here is a comment that I preach in the major capstone course and, somewhat surprisingly, must include in many referee reports:
The authors do not provide a summary of the data. Instead, table 1 is the regression analysis. In order to assess data quality and better understand the results of the regression analysis a univariate analysis of the dependent and independent variables is needed. One of these tables should include trip responses under each scenario.
The univariate tables can be the most interesting part of a paper. They lay out the potential relationships amongst the variables and make clear just what is included in the empirical model. Plus, when you don't include these numbers it looks like you are hiding something (e.g., crappy data). Here is the abstract from some recommended reading (it is on my reading list for senior capstone students):
A critical objective for many empirical studies is a thorough evaluation of both substantive importance and statistical significance. Feminist economists have critiqued neoclassical economics studies for an excessive focus on statistical machinery at the expense of substantive issues. Drawing from the ongoing debate about the rhetoric of economic inquiry and significance tests, this paper examines approaches for presenting empirical results effectively to ensure that the analysis is accurate, meaningful, and relevant for the conceptual and empirical context. To that end, it demonstrates several measurement issues that affect the interpretation of economic significance and are commonly overlooked in empirical studies. This paper provides guidelines for clearly communicating two distinct aspects of “significance” in empirical research, using prose, tables, and charts based on OLS, logit, and probit regression results. These guidelines are illustrated with samples of ineffective writing annotated to show weaknesses, followed by concrete examples and explanations of improved presentation.
Source: Miller, Jane E., and Yana van der Meulen Rodgers. "Economic importance and statistical significance: Guidelines for communicating empirical research." Feminist Economics 14, no. 2 (2008): 117-149.
I just submitted a review with both of these comments. So much for single blind peer review.
