Peer review is at the heart of the scientific method. Its philosophy is based on the idea that one’s research must survive the scrutiny of experts before it is presented to the larger scientific community as worthy of serious consideration. Reviewers (also known as referees) are experts in a particular topic or field. They have the requisite experience and knowledge to evaluate whether a study’s methods are appropriate, results are accurate, and the authors’ interpretations of the results are reasonable. Referees are expected to alert the journal editor to any problems they identify, and make recommendations as to whether a paper should be accepted, returned to the authors for revisions, or rejected. Referees are not expected to replicate results or (necessarily) to be able to identify deliberate fraud. While it’s by no means a perfect system (see, for example, the rising rates of paper retractions), it is still the best system of scientific quality control that we have. In fact, it’s such a central part of the scientific process that one can often identify questionable scientific findings by their authors’ objections to the rigor of review (and attempts to circumvent it by a variety of methods, including self-publication).
Yet the quality of this system is ultimately dependent on the quality of the reviewers themselves. Many graduate programs don’t explicitly teach courses on how to review papers. Instead, a young scientist may learn how to review a paper under the guidance of his or her mentor, through journal clubs, or simply through trial and error. I’ve been thinking about this topic a lot, and decided to put together a set of guidelines for young scientists. In doing so, I also hope to help non-scientists understand a bit more about the process.
I intend for this to be an evolving post, so I ask for my colleagues’ help in improving it. At the end the post, I’m assembling a list of resources for further reading. If you have any suggestions or resources, please send them to me and I’ll add them.
Ethical considerations
Here are some important practices to keep in mind as you conduct the review:
1. Expertise: Ask yourself honestly whether this paper falls within the scope of your expertise. If it falls too far outside of your discipline and knowledge, it’s better to leave the review to someone else. If you decline to review (for whatever reason), it’s courteous to recommend several alternate referees to the editor.
2. Timeliness: It’s important that you make an effort to return a review within the specified deadline. People’s research careers may be depending on how quickly their papers are published. Take into consideration your schedule when you’re deciding whether to review a paper. If you’re a relatively young researcher, you may need to put in additional time in order to familiarize yourself with any background literature and methods that might be new to you. This isn’t a small thing, especially when balanced with all your other duties (coursework, teaching, research, writing). And keep in mind that complicated papers take more time to evaluate. For example, it took me about 8 or 10 hours (spread out over several days) to do my most recent review of a particular manuscript, and another 4 hours to review the revised version. If you don’t think you can commit enough time to the review in order to make the deadline, you should decline.
3. Take it seriously:Peer review is at the heart of the scientific process, and in order for the process to work, the review must be rigorous. Don’t just sign off on a paper or do a slapdash job; read it critically (see my suggestions for how to do so below). Your review should ideally help the authors to improve the quality of their manuscript, and contribute to the overall quality of the journal.
4. Avoid bias: Absolutely do not review a manuscript if you have a strong feeling (positive or negative) about any of the authors. Absolutely do not review a manuscript with any intention of reaping personal or professional benefit from it. Be aware that several studies have revealed implicit biases (such as gender bias) in peer review. Abstracts with female senior authors are systematically more poorly reviewed than abstracts with male senior authors (Knobloch-Westerwick et al. 2013), and “all articles with women in dominant author positions receive fewer citations than those with men in the same positions.” (Larivière et al. 2013)
To avoid biases of all kinds, make a concerted, deliberate effort to always review the manuscript in front of you, not the authors.
5. Don’t be intimidated: Remember that reviewing is a place where the implicit hierarchy of academia should not apply, ever. The authors of the manuscript might be the biggest names in the field, but that doesn’t automatically make them right.
6. Review anonymously? There have been some calls in discussions of peer review for doing away with anonymity of referees, as anonymity is seen by some as facilitating bullying. In fact, the British Medical Journal (BMJ) requires all reviewers to be identified. In a study of the effect of open peer review on the quality of reviews, it was suggested that “if reviewers have to sign their reviews they may put more effort into their reviews and so produce better ones “, and also that “ open peer review should increase both the credit and accountability for peer reviewing, both of which seem desirable.” (van Rooyen et al. 1999). In contrast, this discussion in Nature Neuroscience recommended the continuation of anonymous review.
As an early career scientist, I feel more comfortable critiquing my colleagues’ work anonymously. But I know several peers who have reached the opposite conclusion. Ultimately, the decision as to whether or not to sign your review is a completely personal one.
7. Respect confidentiality: Don’t talk about the manuscript, its results, or its methods with outsiders. Don’t use information from the manuscript prior to its publication. Don’t communicate with the authors about their manuscript. All thoughts and comments on it should only go to the editor.
How to review critically, but fairly
You should be an expert at this by now, because you’ve been reading and thinking about papers nonstop since the beginning of graduate school. The process is basically the same for reviewing, except that a reviewer must be even more thorough. Here are a few tips, divided into four topics (if
you want some more suggestions on how to read critically, my post on how to read and understand the scientific literature applies to scientists as much as non-scientists). I suggest you take notes constantly as you go through the review process. Draft your response only after you’ve read the entirety of the submission (including Supplementary Online Information) several times! Avoid being unnecessarily harsh or abusive; your criticism should be constructive in tone.
1) The Introduction/Background
Some questions to ask:
An approach to critically evaluating the Introduction/Background portion of a manuscript.
Things to watch out for in this section:
-An unoriginal, uninteresting, or irrelevant study. Ask whether the questions the authors are addressing really advance the field in a meaningful way.
-A paper which does not include all appropriate references, or which inappropriately emphasizes some. I suggest identifying the specific question and doing a literature search on your own, as if YOU were writing the paper. Then compare your list of references to the authors’. Are you familiar with all of them? (This approach has tangential benefits, as it will help you become better acquainted with the literature of your field). If the authors cite certain studies with which you’re not familiar, read them! See if you agree with how they’re discussed. If the authors haven’t cited studies that they should have, make a note of it, and include it in your review. And yes, it’s perfectly fine to suggest that they cite your own papers, if it’s appropriate (the flip side: don’t be “that reviewer” who demands that everyone cite him/her, or agree with his/her pet hypotheses. This process isn’t about you).
2) Approach/Methods
Some questions to ask:
An approach to critically evaluating the Approach/Methods portion of a manuscript
Don’t hesitate to go read outside sources on how to conduct an experiment or how other labs use them. Don’t hesitate to (while still respecting the confidentiality of the manuscript) ask other people for help understanding methods in a general sense. You need to be confident in your assessment of whether the authors did the experiments correctly. If you’re not, tell the editor privately and request an additional expert to review that portion of the manuscript.
Keep in mind that you have every right to ask the authors do a different experiment, or to modify the way they carry out an existing one. Remember that doing so will take time, effort, and money, so don’t do this lightly. (And remember that under no circumstances should you even think of doing this in order to slow a lab down so that your project can be published first. That’s EXTREMELY unethical.).
3) Interpreting the research
Some questions to ask:
An approach to critically evaluating the Results/Interpretation/Discussion portion of a manuscript
Absolutely feel free to ask for more results, or better ways of presenting them. If you believe that the data require the review of a statistical expert (or another kind of expert), recommend it to the editor. If you believe material in the Supplementary Online Information section would be more useful in the main body of the paper (or vice versa), recommend it!
4. Some final considerations
By this point, you will probably have read the entirety of the paper several times. Is the writing clear and free of grammatical errors? If not, request additional editing before publication (it isn’t necessarily your job to point out specific grammatical errors, though you are free to do so). Have the authors specified a mechanism by which they will make raw data from their experiments available? Have they specified how they will protect the identity of their human subjects (particularly a concern in genomics research)? Are there any additional ways in which the paper could be improved? Include these considerations in your write up to the editor.
Concluding thoughts
As you can see, reviewing can be a very difficult task, but you definitely will improve with practice! Journal clubs are an excellent way to improve your critical reading skills, as well as help from your mentor. I’ve found the best way of improving my review abilities is to actually go through the process as an author. Remember that (minus the odd antagonistic person), referees are there to help you improve your work. Listen to and learn from them!
The scientific community demands a lot from our reviewers. Reviewing manuscripts may not give you obvious immediate benefits, but understand that doing so is a good way of keeping abreast of cutting edge research in your discipline. Going through the process makes you better at critical reading and thinking, and allows you to give back to your field in a very important way. I view it as a sort of civic obligation to the broader scientific community. Your contribution as a referee helps keep the scientific process as fair as possible.
Many thanks to Dr. Elizabeth Wagner for her helpful thoughts and resources on this subject.
References and useful resources
Wager et al. 2002. “How to survive peer review” London: BMJ Book.http://www.bmj.com/sites/default/files/attachments/resources/2011/07/wager.pdf
Committee on publication ethics: Guidelines for Peer Reviewers by Irene Hames on behalf of COPE Council, March 2013, v. 1.http://publicationethics.org/files/u7140/Peer%20review%20guidelines.pdf
van Rooyen et al. 1999. Development of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts. Journal of Clinical Epidemiology 52(7) 625-629. (http://www.sciencedirect.com/science/article/pii/S0895435699000475)
Nicholas, KA and Gordon, WS. 2011 “A Quick Guide to Writing a Solid Peer Review” EOS 92 (28): 233-240. (http://onlinelibrary.wiley.com/doi/10.1029/2011EO280001/pdf)
Parberry, I. “A guide for new referees in theoretical computer science.”http://www.cs.utexas.edu/~dahlin/professional/parbery-referee.pdf
Moher D and Jadad AR. “How to peer review a manuscript”http://www.bmj.com/sites/default/files/attachments/resources/2011/07/moher.pdf
Nature’s checklist for authors:http://www.nature.com/authors/policies/checklist.pdf
PRISMA’s resources for systematic reviews and meta-analyses:http://www.prisma-statement.org/
STROBE’s resources for epidemiology studies: http://www.strobe-statement.org/index.php?id=available-checklists
Nature Neuroscience’s guide for reviewers (http://www.nature.com/neuro/referees/index.html), and “Striving for excellence in peer review” (http://www.nature.com/neuro/journal/v12/n1/full/nn0109-1.html)
Petrovečki, M. The role of statistical reviewer in biomedical scientific journal. Biochemia Medica 2009;19(3):223-30.http://dx.doi.org/10.11613/BM.2009.020
Gould, T. H. P. (2010). Scholar as e-publisher: The future role of [anonymous] peer review within online publishing. Retrieved from http://krex.ksu.edu
van Rooyen et al. 1999. Effect of open peer review on quality of reviews and on
reviewers’ recommendations: a randomised trial. BMJ 318 pp 23-27
Wager et al.2006. Are reviewers suggested by authors as good as those chosen by
editors? Results of a rater-blinded, retrospective study. BMC Medicine 2006, 4:13 (http://www.biomedcentral.com/1741-7015/4/13)
Please feel free to suggest additional resources, or your thoughts on how to improve this post in the comments below, or by emailing me at jenniferraff (at) utexas (dot) edu. Thanks in advance!
