Apparently, some psychology journals are not allowing p-values (or asterisks, I guess) on regression coefficients. Noah Smith:
... Significance tests results shouldn't be used in a vacuum - to do good science, you should also look at effect sizes and goodness-of-fit. There is a culture out there - in econ, and probably in other fields, that thinks "if the finding is statistically significant, it's interesting." This is a bad way of thinking. Yes, it's true that most statistically insignificant findings are uninteresting, but the converse is not true. For something to be interesting, it should also have a big effect size. The definition of "big" will vary depending on the scientific question, of course. And in cases where you care about predictive power in addition to treatment effects, an interesting model should also do well on some kind of goodness-of-fit measure, like an information criterion or an adjusted R-squared or whatever - again, with "well" defined differently for different problems. Yes, there are people out there who only look at p-values when deciding whether a finding is interesting, but that just means they're using the tool of p-values wrong, not that p-values are a bad tool. ...
Fortunately I see a few signs of a backlash-to-the-backlash against significance testing. In 2005 we had a famous paper that used simulations to show that "most published research findings [should be] false". Now, in 2015, we have a meta-analysis showing that the effect of p-hacking, though real, is probably quantitatively small. In addition, I see some signs on Twitter, blogs, etc. that people are starting to get tired of the constant denunciation of significance testing - it's more of a hipster trend than anything. Dissing p-values in 2015 is a little like dissing macroeconomics in 2011 -
via noahpinionblog.blogspot.com
Most economics papers that I read have sample sizes large enough (and not too large) that the p-values are meaningful. In the area of nonmarket valuation, where we estimate willingness to pay and consumer surplus, effect sizes are always assessed so I am not very familiar with that part of the problem. But when I teach senior seminar (the last time was Fall 2013) I emphasize effect sizes. Here is the recommended outline from the paper guidelines:
Paper Outline:
- Introduction - describe the economic problem that you are addressing (rewrite your paper proposal). The last paragraph of the introduction should begin with “the purpose of this paper is to ….” The introduction should be about 1 page in length.
- Literature Review – The purpose of the literature review is to learn as much as you can from the efforts and work of others; it prevents duplication and/or identifies replication needs; it identifies the frontier of knowledge and the potential research contribution of your paper; it provides ideas and directions. In the literature review you should analyze, compare and contrast the papers in the annotated bibliography. Make sure you identify a gap in the literature. Don't cut and paste the five paragraphs of you annotated bibliography. The literature review should be between 1 and 2 pages.
- Model - describe the empirical models that you intend to estimate and the expected effect of the independent variable on the dependent variable using economic theory
- Data - describe the source of your data, what you did with it (i.e., transformations of variables) and univariate statistics (Tables 1 and 2). ...
- Results – describe the statistical results in some detail (Table 3)
- Discussion – describe the economic importance of your results; interpret marginal effects, elasticities, etc.; employ at least one bar chart or graph
- Conclusions - summarize your findings, how they compare with earlier literature, and address any policy implications and unresolved issues.
- Tables - Table 1. Variable descriptions; Table 2. Summary statistics; Table 3. Regression models; Table 4. Economic impacts
- Figures - at least one bar chart illustrating economic importance (for categorical variables), or at least one graph illustrating economic importance (for continuous variables)
- References
What am I missing?
