Figure 1 Activating the _preview style
Wichor M. Bramer, Jelena Milic, MD, Frans Mast, PhD
doi: http://dx.doi.org/10.5195/jmla.2017.111
Received 01 June 2016: Accepted 01 August 2016
An important task in conducting a systematic review is reading titles and abstracts of the retrieved references, which often number in the thousands, to determine which articles meet the predefined inclusion criteria. In the past, this was performed by manually scanning through large stacks of printed titles and abstracts, followed by face-to-face meetings to discuss which references should be included. Today, the workflow of the review process is more streamlined by using computer software.
Several specialized solutions for this process are available, most notably free or subscription-based online tools such as Covidence, DistillerSR, or Rayyan. The Cochrane collaboration uses its own tool called ReviewManager. A survey in 2013 showed that more than half of all systematic reviewers used EndNote software [1]. This dominance on the market is likely to increase as sales of the second most popular tool, Reference Manager, have ceased, and its website now advises users to switch to EndNote. Many reviewers use Microsoft Excel. Some libraries have even created specialized Excel workbooks to document the process in much detail [2]. A method for the inclusion process using EndNote is described by King et al. [3], but this process is rather complicated and time-consuming.
This paper describes the logistics of a method to perform the title and abstract screening, verdict assignment, and comparisons of results among multiple reviewers in EndNote. The process is blinded; all reviewers work in their own EndNote files, and after the individual inclusion and exclusion processes, the verdicts of the different reviewers are compared. The method described here can be performed much faster than comparable methods.
The method consists of several steps. First, a custom style should be installed for easy abstract scanning. Second, a field should be added to show the reviewer’s name in the Library window. Third, for each systematic review, custom groups are made in the EndNote library for included and excluded references. Reviewers drag articles to the group corresponding to their verdicts. In the last steps for the final comparison of the verdicts, the included references of all reviewers are combined into one EndNote library and de-duplicated. References found as duplicates are included by both reviewers and are selected for full-text review, the non-duplicate references can then be discussed for inclusion or exclusion.
The steps below are written for EndNote X7 for Windows. EndNote is also available in a version for Apple Mac computers, but some of the menu items will appear on different places. The authors have added footnotes that guide Mac users as much as possible.
A custom-made style (named _preview) facilitates easier reviewing of the titles and abstracts.
Visit http://bit.ly/emcendnote
Open the zip file
Double-click on the file_preview.ens (it will open in EndNote)*
In EndNote, click File > Save As
Remove the text “copy” from the file name, and click Save†
Close the style using the cross in the top right corner‡
To activate the _preview style, open the drop-down menu (called Select Bibliographic Output Style) in the top left part of the screen (Figure 1)§
Click on Select another style, scroll to the top, select _preview, and click Choose
The abstract of the selected reference will be displayed in the preview tab in the Tab pane
|
|
||
|
Figure 1 Activating the _preview style |
||
When comparing the included references between reviewers, a special field will be used to document the name of the reviewer who included a certain reference. To be able to view this in the Library window, this field has to be added to the preference settings.
Go to Edit > Preferences > Display Fields
In column 8 (or an alternative field that is not often used), under Field, select Custom 4
In the same column, under Heading, type Reviewer
In the first reviewing round, two reviewers independently read titles and abstracts to decide whether a reference is potentially relevant to the review. We propose the creation of two group sets: Includes and Excludes, each with, at this stage, just one subgroup (Figure 2).
Right-click on My Groups, and select Create Group Set
Type Excludes, and hit Enter
Right-click on Excludes, and select Create Group
Type title/abstract, and hit Enter
Repeat the process for Group Set Includes with Includes group
|
|
||
|
Figure 2 Groups for Includes and Excludes |
||
After the groups for inclusion and exclusion have been made, two copies of these files are created (adding the name of the reviewer to the file name) and distributed to the reviewers. All reviewers will work in their own copies of the EndNote file.
The standard group named Unfiled contains references not yet assigned to one of the other groups. When starting the screening phase, the number of references in Unfiled will be equal to that in All References. Each reviewer reviews the relevance of references in Unfiled one by one, based on the title and/or abstract.
Broaden the title field in the Library window (Figure 2) by dragging the column break between the columns of Title and Journal to the right until the Title column reaches an appropriate width
Review the titles one by one in the Library window until a potentially relevant title is reached, without yet assigning references to the groups
Click on the relevant title, and read the abstract in the preview tab of the Tab pane (Figure 2)
If the abstract is irrelevant, continue reading titles in the Library window
If the abstract is relevant, select the article directly above that relevant article in the Library window, and press Ctrl+Shift+Home to select all references above it. Drag all of these articles into the Excludes > Title/Abstract group
Next, drag the top reference in the Library window (which is the reference to be included, as irrelevant references were removed from Unfiled) to the Includes > Includes group
Repeat this process until all references are filed and the Unfiled group is emptied
Go to the folder Includes in the EndNote library of reviewer 1, and click on one of the references
Go to Tools > Change/Move/Copy Fields
In Custom 4, select Insert after field’s text, type the first name of reviewer 1, and click on OK in three pop-up screens
Open the EndNote library of reviewer 2 without closing that of reviewer 1
Go to the Includes group in the library of reviewer 2; select all references in that group, right click on one of them, select Copy reference to, and select the file screened by reviewer 1
Go to Window, and select the file by reviewer 1
Go to the Unfiled group, and mark the records in that group with the name of reviewer 2 (as described above in steps 2–3)
Drag the references from Unfiled into the Includes group
Check the settings for de-duplication (Edit > Preferences > Duplicates); at least Author, Year, Title, and Secondary Title (Journal) should be selected
Go to the group Includes, select a random reference, and go to References > Find Duplicates; in the detailed comparison screen, click Cancel; then press Delete on the keyboard to remove the duplicate references
Right click in the group set Includes, select Create a group, and name it Definite Includes
Select all references in the Duplicate References group, and drag those to the Definite Includes group
Go to the Definite Includes group, and mark the records in that group with the name of reviewer 2 (as described above in steps 2–3)
Right click on the Includes group > Includes, and select delete group
Have the two reviewers discuss the articles currently in the Unfiled group
After consensus is reached, drag the references one by one to the appropriate group until Unfiled is empty
In the second round of screening, full texts of the included titles and abstracts need to be reviewed. Custom groups can be used to distinguish between various reasons for exclusion, and articles can be assigned to specific groups for certain sub-questions. All reviewers should again work in their own copies of this library. After reading all articles, each reference in the library should be discussed in detail; therefore, no automatic comparison should be used. The steps in this round are more laborious, differ per research topic, and can hardly be generalized and optimized. Therefore, we do not describe in detail how this process can be executed.
We advise against the use of a “Doubt” or “Maybe” group for articles for which it is not yet clear whether they should be included. We recommend that in cases of doubt, the article should be added to the folder Includes. If the second reviewer also has doubts about the relevance or decided to include the article, the full text should be used for final judgment. If the second reviewer excludes the article, it is an item for discussion.
The process of reading titles and abstracts for inclusion and exclusion is often considered time consuming, and the number of abstracts that can be read per hour is estimated at 120 by the Cochrane Handbook [4]. A recent study recorded the time needed for certain steps in the systematic review process [5]. Upon request, the authors informed us that, using specialized screening software, the median number of articles that could be reviewed per hour was 68. Another recent study estimated the time needed to screen 1 record based on title and/or abstract at 1 minute [6]. A survey among review authors at Erasmus MC, to whom we had sent an earlier draft of this article, reported that the median number of titles and/or abstracts reviewed per hour with the present method was 308, with a maximum of 675.
We found that the speed of the process increases when reviewers do not document the specific reasons for excluding references during the title/abstract screening phase. It is often clear that an article is not relevant to the topic, but to determine the exact reason (or very often multiple reasons) is very time consuming and unnecessary. According to PRISMA guidelines, reasons for exclusion, with the number of articles for each reason, should be given only in the full-text screening phase [7]. We found that researchers who meticulously documented the reasons for exclusion in the first round and those who had used other software such as Microsoft Excel reported much slower rates (median of sixty minutes) and often later regretted their decision to do so. Also, using specialized programs or online systematic review management systems such as Covidence or DistillerSR unnecessarily complicate and delay the process because each abstract has to be assigned to categories individually, in contrast to this method’s bulk assignment of nonrelevant articles.
The unpublished internal document in which staff first described the method is based on a similar document for Reference Manager, written by Hans Ket of the VU Medical Library, Amsterdam, the Netherlands. We thank former colleague Louis Volkers for developing the first _preview output style, which is used in this method to ease accessing the abstract of retrieved references. We also thank Gerdien de Jonge of the Medical Library of Erasmus MC for her input in the method and critical review of earlier drafts of this article.
1 Lorenzetti D, Ghali W. Reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. BMC Med Res Methodol. 2013;13(1):141.
2 VonVille HM. Excel workbooks for systematic reviews & corresponding handouts [Internet]. University of Texas [cited 19 Apr 2016]. <http://libguides.sph.uth.tmc.edu/excel_SR_workbook>.
3 King R, Hooper B, Wood W. Using bibliographic software to appraise and code data in educational systematic review research. Med Teacher. 2011;33(9):719–23.
4 Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. Wiley Online Library; 2008.
5 Wang Z, Asi N, Elraiyah TA, Abu Dabrh AM, Undavalli C, Glasziou P, Montori V, Murad MH. Dual computer monitors to increase efficiency of conducting systematic reviews. J Clin Epidemiol. 2014 Dec;67(12):1353–7.
6 Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews. 2016;5(1):1–13.
7 Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009 Oct;62(10):1006–12.
*For Mac users: The file must be downloaded and saved locally before it can be opened. ( Return to Text )
†For Mac users: The text “copy” will not appear in the file name. ( Return to Text )
‡For Mac users: The cross appears in the top left corner. ( Return to Text )
§For Mac users: At the bottom right of the screen is a preview pane. The style selection menu appears at the top of the pane. ( Return to Text )
Wichor M. Bramer, w.bramer@erasmusmc.nl, Biomedical Information Specialist, Medical Library, Erasmus MC–Erasmus University Medical Centre, Rotterdam, The Netherlands
Jelena Milic, MD, j.milic@erasmusmc.nl, Department of Epidemiology, Erasmus MC, University Medical Centre, Rotterdam, The Netherlands
Frans Mast, PhD, f.mast@erasmusmc.nl, Medical Library, Erasmus MC, University Medical Centre, Rotterdam, The Netherlands
Articles in this journal are licensed under a Creative Commons Attribution 4.0 International License.
This journal is published by the University Library System of the University of Pittsburgh as part of its D-Scribe Digital Publishing Program and is cosponsored by the University of Pittsburgh Press.
Journal of the Medical Library Association, VOLUME 105, NUMBER 1, January 2017
