Katherine G. Akers, PhD
doi: http://dx.doi.org/10.5195/jmla.2017.100
As a peer reviewer for a journal, you have an important role to play. You help the editor sift through the myriad of submissions, evaluating the strengths and weaknesses of manuscripts and thereby helping make decisions about which works are important and rigorous enough to be shared with our community of library researchers and practitioners. As in most fields of study, however, librarians usually do not receive formal training in how to review a manuscript. Instead, we may learn how to be a peer reviewer by receiving reviews of our own manuscripts that we have submitted to journals or by picking up general tips and tricks gleaned from our intimate understanding of the scholarly publishing process. Not surprisingly, therefore, peer-reviewing skills are unevenly distributed across librarians, and even experienced peer reviewers may persist in wondering whether they are successfully accomplishing this task.
If you are relatively new to scholarly publishing and peer reviewing, you may feel a twinge of doubt the first couple of times you are asked to review a manuscript. You may fear that you will not be able to come up with at least one solid criticism of the manuscript that now lies in your hands. Rest assured, however, that this is quite unlikely.
Think of yourself as an external consultant [1]. Realize that you are not expected to be an expert in all aspects of the study described in the manuscript. For instance, you may be well acquainted with the study’s subject matter but not its methodology. This is okay. Comment on what you can, and chances are that the other reviewers of the manuscript will be able to point out issues that you missed (and that you will be able to point out issues that the other reviewers missed). Remember that you have a unique perspective on the work. By virtue of being a person who did not conduct the research or write the manuscript—by having intellectual and emotional distance from the work—you are in an ideal position to spot methodological errors or inconsistencies in reporting, to determine whether the descriptions of the methodology and results make sense, and to judge whether the manuscript advances the knowledgebase or the practice of librarianship.
If you are new to the library profession, realize that your inexperience is not necessarily a detriment to your peer-reviewing ability. In fact, for medical journals, studies consistently show that younger reviewers or reviewers with less professional experience write higher quality reviews than those who are older or have more experience [2–5]. Although it is not yet known why younger reviewers write better reviews, it may be because they are more enthusiastic, less couched in their own methods and opinions, or less burdened by other professional responsibilities.
Some common advice given to peer reviewers is to read the manuscript at least twice [6, 7]. The purpose of the first reading is to get an overall sense of why the study was performed, how it was performed, and what its main findings are. As you read through the manuscript a second time, pay attention to the details of the methodology, the data reported in the text and shown in tables or figures, and the logic of the authors’ arguments and conclusions.
Take notes while you read the manuscript, and then construct those notes into a carefully written review. Discuss both the strengths and weaknesses of the manuscript. When you find a weakness, suggest a concrete way to overcome the shortcoming, if possible: “It’s the easiest thing in the world to poke holes into something. It is usually much harder to suggest how to fix them” [8]. It is usually helpful to the editor and authors if you conceptually organize your review in some manner. For instance, your comments could be grouped into major versus minor points or separated by sections (e.g., Introduction, Methods, Results, and Discussion). Enumerating your comments or using bullet points also helps the editor and authors tease apart multiple concerns. Although identifying individual grammatical errors is generally not advised, it is appropriate to mention if you think the manuscript is poorly written overall or in particular sections. After completing the first draft of your review, read it again to ensure that it is written in a professional tone. Be polite but confident; do not be afraid to point out the manuscript’s limitations.
To get started reviewing a manuscript, consider the following questions.
Do the authors provide a compelling rationale for why they conducted the study?
Do the authors clearly describe the purpose of the study and/or state their hypothesis?
Are the methods fully and clearly described?
Is the methodology sound? Are there potential sources of bias?
Do the authors utilize objective measures (e.g., of impact, use, success) when possible?
Are the results reported in the text consistent with the data shown in the figures or tables?
Are the statistical analyses performed and reported appropriately?
Do the authors draw clear conclusions based on the results (as opposed to simply restating the results)?
Are the conclusions justified by the data (or do they overreach the data)?
Do the authors explain how their findings advance the knowledgebase and practice of the field?
Do the figures and tables clearly convey the information?
Are all of the figures and table necessary (or, conversely, do you recommend additional figures or tables)?
Would the manuscript interest the journal’s readership?
Is the writing straightforward and to-the-point?
Are there areas of the text that the authors should clarify, elaborate upon, or omit?
Are the authors missing any pertinent references or body of literature?
There is no set recommendation for how long reviewing a manuscript should take. Although some reviewers may spend 8 or more hours reading the manuscript and writing a review [9], a study of a public health journal shows that completing a review takes 2.7 hours on average [10]. Furthermore, spending more time performing a review does not necessarily result in a stronger review. Whereas one study reports that spending 4 hours or more is associated with higher quality reviews [3], another study reports that there is no further improvement in review quality after 3 hours [4].
Although writing a review need not take a great amount of time, it is generally true that longer reviews are better than very brief reviews. More specifically, a good, comprehensive review should typically be between one-half to two pages in length [11], depending on the complexity and quality of the manuscript. Some sound advice is to “be loquacious” [8]. When you make a suggestion to the authors, explain your reasoning behind that suggestion. Trying to interpret a reviewer’s vague comments can be extremely frustrating for authors. On the other hand, clearly explaining your thinking makes it much more likely that the authors will be able to accurately and adequately address your concerns.
Finally, if you find a “fatal flaw” in the manuscript, such as an error in logic or the use of an inappropriate research design or approach that cannot be remedied by collecting additional data or rewriting the manuscript, it is unnecessary to prepare a long list of minor comments; rather, simply communicate your major concerns [11].
Conducting peer review is a service to our academic and professional community. Because authors and readers are usually unaware of the identity of reviewers, peer reviewing is mostly a thankless job. However, some recent initiatives aim to provide scholarly credit for peer review (although this is not without controversy [12]). In particular, Publons is a social media site that can verify your review assignments, showcase your peer review contributions, and allow you to keep a record of your reviewing activity for tenure or promotion applications or annual performance reviews [13].
Despite radical changes in scholarly communication, peer review is still considered a necessary process that increases the quality of published research [14]. Speaking on behalf of the editorial team at the Journal of the Medical Library Association, we sincerely thank our peer reviewers, who take great care in moderating and strengthening the discourse on the research and practice of health sciences librarianship.
1 McKarney L. Peer-review techniques for novices. Science Careers [Internet]. American Association for the Advancement of Science [20 Apr 2011; cited 4 Oct 2016]. <http://www.sciencemag.org/careers/2001/04/peer-review-techniques-novices>.
2 Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med. 2007;4(1):e40.
3 Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993 Aug;8(8):422–8.
4 Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998 Jul 15;280(3):231–3.
5 Kliewer MA, Freed KS, DeLong DM, Pickhardt PJ, Provenzale JM. Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American Journal of Roentgenology. Am J Roentgenol. 2005 Jun;184(6):1731–5.
6 Allen TW. Peer review guidance: how do you write a good review? J Am Osetopath Assoc. 2013 Dec;113(12):916–20.
7 Nicholas KA, Gordon W. A quick guide to writing a solid peer review. Eos. 2011 Jul;92(28):233–40.
8 Lucey B. Peer review: how to get it right – 10 tips. Higher Education Network [internet]. The Guardian [27 Sep 2013; cited 4 Oct 2016]. <https://www.theguardian.com/higher-education-network/blog/2013/sep/27/peer-review-10-tips-research-paper>.
9 Raff J. How to become good at peer review: a guide for young scientists. Violent Metaphors [Internet]. [13 Dec 2013; cited 4 Oct 2016]. <https://violentmetaphors.com/2013/12/13/how-to-become-good-at-peer-review-a-guide-for-young-scientists/>.
10 Yankauer A. Who are the peer reviewers and how much do they review? JAMA. 1990 Mar 9;263(10):1338–40.
11 Brand RA. Reviewing for clinical orthopaedics and related research. Clin Orthop Relat Res. 2012 Sep;470(9):2622–5.
12 Crotty D. The problem(s) with credit for peer review. Scholarly Kitchen [Internet]. [25 Jun 2015; cited 4 Oct 2016]. <https://scholarlykitchen.sspnet.org/2015/06/17/the-problems-with-credit-for-peer-review/>.
13 Publons: Get credit for peer review [Internet]. [cited 4 Oct 2016]. <https://publons.com>.
14 Mulligan A, Hall L, Raphael E. Peer review in a changing world: an international study measuring the attitudes of researchers. J SIST. 2013 Jan;64(1):132–61.
Katherine G. Akers, PhD, jmla@journals.pitt.edu, Interim Editor-in-Chief, Journal of the Medical Library Association
Articles in this journal are licensed under a Creative Commons Attribution 4.0 International License.
This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press.
Journal of the Medical Library Association, VOLUME 105, NUMBER 1, January 2017