Peer review is at the heart of the scientific method. Its philosophy is based on the idea that one’s research must survive the scrutiny of experts before it is presented to the larger scientific community as worthy of serious consideration. Reviewers (also known as referees) are experts in a particular topic or field. They have the requisite experience and knowledge to evaluate whether a study’s methods are appropriate, results are accurate, and the authors’ interpretations of the results are reasonable. Referees are expected to alert the journal editor to any problems they identify, and make recommendations as to whether a paper should be accepted, returned to the authors for revisions, or rejected. Referees are not expected to replicate results or (necessarily) to be able to identify deliberate fraud. While it’s by no means a perfect system (see, for example, the rising rates of paper retractions), it is still the best system of scientific quality control that we have. In fact, it’s such a central part of the scientific process that one can often identify questionable scientific findings by their authors’ objections to the rigor of review (and attempts to circumvent it by a variety of methods, including self-publication).
Yet the quality of this system is ultimately dependent on the quality of the reviewers themselves. Many graduate programs don’t explicitly teach courses on how to review papers. Instead, a young scientist may learn how to review a paper under the guidance of his or her mentor, through journal clubs, or simply through trial and error. I’ve been thinking about this topic a lot, and decided to put together a set of guidelines for young scientists. In doing so, I also hope to help non-scientists understand a bit more about the process.
The guidelines are mostly spot-on for economists, although a bit too thorough (I still cringe at the thought of the 7 page report recommending rejection that I received from JEEM many years ago). My opinion is that your report should typically be between 500 and 1000 words (1-2 single spaced pages, Times New Roman font, 1 inch margins).
Hat tip: Retraction Watch