Errors in search strategies used in systematic reviews and their effects on information retrieval
DOI:
https://doi.org/10.5195/jmla.2019.567Keywords:
Systematic Reviews, Information Retrieval, Search StrategiesAbstract
Objectives: Errors in search strategies negatively affect the quality and validity of systematic reviews. The primary objective of this study was to evaluate searches performed in MEDLINE/PubMed to identify errors and determine their effects on information retrieval.
Methods: A PubMed search was conducted using the systematic review filter to identify articles that were published in January of 2018. Systematic reviews or meta-analyses were selected from a systematic search for literature containing reproducible and explicit search strategies in MEDLINE/PubMed. Data were extracted from these studies related to ten types of errors and to the terms and phrases search modes.
Results: The study included 137 systematic reviews in which the number of search strategies containing some type of error was very high (92.7%). Errors that affected recall were the most frequent (78.1%), and the most common search errors involved missing terms in both natural language and controlled language and those related to Medical Subject Headings (MeSH) search terms and the non-retrieval of their more specific terms.
Conclusions: To improve the quality of searches and avoid errors, it is essential to plan the search strategy carefully, which includes consulting the MeSH database to identify the concepts and choose all appropriate terms, both descriptors and synonyms, and combining search techniques in the free-text and controlled-language fields, truncating the terms appropriately to retrieve all their variants.References
Eden J, Levit L, Berg A, Morton S, eds. Finding what works in health care: standards for systematic reviews. Washington, DC: National Academies Press; 2011. 340 p. DOI: http://dx.doi.org/10.17226/13059.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009 Oct;62(10):e1–34.
Lefevre C, Manheimer E, Glanville J. Searching for studies. In: Higgins J, Green S, eds. Cochrane handbook for systematic reviews of interventions [Internet]. Version 5.1.0. West Sussex, England, UK: John Wiley & Sons; 2011. p. 95–150 [cited 29 May 2018]. <http://handbook-5-1.cochrane.org/>.
Buckland M, Gey F. The relationship between recall and precision. J Am Soc Inf Sci. 1994 Jan;45(1):12–9.
McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005 Jan;93(1):74–80.
Lane D, Dykeman J, Ferri M, Goldsmith CH, Stelfox HT. Capture-mark-recapture as a tool for estimating the number of articles available for systematic reviews in critical care medicine. J Crit Care. 2013 Aug;28(4):469–75.
Wieland LS, Robinson KA, Dickersin K. Understanding why evidence from randomised clinical trials may not be retrieved from MEDLINE: comparison of indexed and non-indexed records. BMJ. 2012 Jan;344(7838):1–10.
Delaney A, Tamás PA. Searching for evidence or approval? a commentary on database search in systematic reviews and alternative information retrieval methodologies. Res Synth Methods. 2018 Mar;9(1):124–31.
Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol. 2006 Oct;59(10):1057–63.
Goossen K, Tenckhoff S, Probst P, Grummich K, Mihaljevic AL. Optimal literature search for systematic reviews in surgery. Langenbecks Arch Surg. 2018 Feb;403(1):119–29.
Harter SP. Online information retrieval. concepts, principles and techniques. San Diego, CA: Academic Press; 1986.
Lancaster FW, Warner AJ. Information retrieval today. Arlington, VA: Information Resources Press; 1993.
Meadow CT, Boyde BR, Kraft DH. Text information retrieval systems. 2nd ed. San Diego, CA: Academic Press; 2002.
Walker G, Janes J. Online retrieval: a dialogue of theory and practice. 2nd ed. Englewood, CO: Libraries Unlimited; 1999.
Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews [Internet]. Oslo, Norway: Campbell Collaboration; 2017 [cited 29 May 2018]. <https://campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html>.
Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, Yao L, Yang K. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014 Sep;67(9):1001–7.
Higgins J, Green S. Cochrane handbook for systematic reviews of interventions [Internet]. Version 5.1.0. Cochrane Collaboration; 2011 [cited 29 May 2018]. <http://handbook-5-1.cochrane.org/>.
Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA. 2014 Sep 10;312(10):999. DOI: http://dx.doi.org/10.1001/jama.2014.9263.
Spencer AJ, Eldredge JD. Roles for librarians in systematic reviews: a scoping review. J Med Libr Assoc. 2018 Jan;106(1):46–56. DOI: http://dx.doi.org/10.5195/jmla.2018.82.
Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015 Jun;68(6):617–26.
Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015 May 4;10(5):1–14.
Zhang L, Sampson M, McGowan J. Reporting of the role of the expert searcher in Cochrane Reviews. Evid Based Libr Inf Pract. 2006;1(4):3–16.
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009 Sep;62(9):944–52.
McGowan J, Sampson M, Lefebvre C. An evidence based checklist for the Peer Review of Electronic Search Strategies (PRESS EBC). Evid Based Libr Inf Pract. 2010;5(1):149.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol. 2016 Jul;75:40–6.
Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009 Jan;97(1):21–9. DOI: http://dx.doi.org/10.3163/1536-5050.97.1.004.
Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008 Aug;61(8):748–54.
Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016 Sep 26;11(9):1–16.
Layton D. A critical review of search strategies used in recent systematic reviews published in selected prosthodontic and implant-related journals: are systematic reviews actually systematic? Int J Prosthodont. 2017 Jan/Feb;30(1):13–21.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007 Mar;4(3):447–55.
Toews LC. Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines. J Med Libr Assoc. 2017 Jul;105(3):233–9. DOI: http://dx.doi.org/10.5195/jmla.2017.246.
Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catalá-López F, Li L, Reid EK, Sarkis-Onofre R, Moher D. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016 May 24;13(5):1–30.
Faggion CM, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. J Clin Epidemiol. 2018 Jun;98:53–61.
Yaylali IE, Alaçam T. Critical assessment of search strategies in systematic reviews in endodontics. J Endod. 2016 Jun; 42(6):854–60.
Russell-Rose T, Chamberlain J. Expert search strategies: the information retrieval practices of healthcare information professionals. JMIR Med Inform. 2017 Oct 2;5(4):e33. DOI: http://dx.doi.org/10.2196/medinform.7680.
Puljak L. If there is only one author or only one database was searched, a study should not be called a systematic review. J Clin Epidemiol. 2017 Nov;91:4–5.
Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. J Med Libr Assoc. 2016 Oct;104(4):284–9. DOI: http://dx.doi.org/10.5195/jmla.2016.141.
Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-Medline in the search for systematic reviews. J Clin Epidemiol. 2011 Jul;64(7):805–7.
US National Library of Medicine. Search strategy used to create the systematic reviews subset on PubMed [Internet]. The Library; 2002 [cited 29 May 2018]. <https://www.nlm.nih.gov/bsd/pubmed_subsets/sysreviews_strategy.html>.
Ho GJ, Liew SM, Ng CJ, Hisham Shunmugam R, Glasziou P. Development of a search strategy for an evidence based retrieval service. PLoS One. 2016 Dec 9;11(12):e0167170. DOI: http://dx.doi.org/10.1371/journal.pone.0167170.
Volpato ESN, Betini M, Puga ME, Agarwal A, Cataneo AJM, Oliveira LD, Bazan R, Braz LG, Pereira JEG, El Dib R. Strategies to optimize MEDLINE and Embase search strategies for anesthesiology systematic reviews. an experimental study. Sao Paulo Med J. 2018 Jan 15;136(2):103–8.
Al-Rifai RH, Ali N, Barigye ET, Al Haddad AHI, Loney T, Al-Maskari F, Ahmed LA. Maternal and birth cohort studies in the Gulf Cooperation Council countries: protocol for a systematic review and narrative evaluation. BMJ Open. 2018 Jan;8(1):e019843. DOI: http://dx.doi.org/10.1136/bmjopen-2017-019843.
Marcolino MS, Oliveira JAQ, D’Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR MHealth UHealth. 2018 Jan;6(1):e23. DOI: http://dx.doi.org/10.2196/mhealth.8873.
Eddens L, van Someren K, Howatson G. The role of intra-session exercise sequence in the interference effect: a systematic review with meta-analysis. Sports Med. 2018 Jan;48(1):177–88.
Ezeamama AE, Bustinduy AL, Nkwata AK, Martinez L, Pabalan N, Boivin MJ, King CH. Cognitive deficits and educational loss in children with schistosome infection. a systematic review and meta-analysis. PLoS Negl Trop Dis. 2018 Jan;12(1):e0005524. DOI: http://dx.doi.org/10.1371/journal.pntd.0005524.
Haniffa R, Isaam I, De Silva AP, Dondorp AM, De Keizer NF. Performance of critical care prognostic scoring systems in low and middle-income countries: a systematic review. Crit Care. 2018 Jan;22(1):18. DOI: http://dx.doi.org/10.1186/s13054-017-1930-8.
Johannsen M, Frederiksen Y, Jensen AB, Zachariae R. Psychosocial predictors of posttreatment pain after nonmetastatic breast cancer treatment: a systematic review and meta-analysis of prospective studies. J Pain Res. 2018 Dec 21;11:23–36.
Cabrini L, Landoni G, Baiardo Radaelli M, Saleh O, Votta CD, Fominskiy E, Putzu A, Snak de Souza CD, Antonelli M, Bellomo R, Pelosi P, Zangrillo A. Tracheal intubation in critically ill patients: a comprehensive systematic review of randomized trials. Crit Care. 2018 Jan;22(1):6. DOI: http://dx.doi.org/10.1186/s13054-017-1927-3.
Britt-Spells AM, Slebodnik M, Sands LP, Rollock D. Effects of perceived discrimination on depressive symptoms among black men residing in the United States: a meta-analysis. Am J Mens Health. 2018 Jan;12(1):52–63.
Gadalla MA, Huang S, Wang R, Norman RJ, Abdullah SA, El Saman AM, Ismail AM, van Wely M, Mol BWJ. Effect of clomiphene citrate on endometrial thickness, ovulation, pregnancy and live birth in anovulatory women: systematic review and meta-analysis. Ultrasound Obstet Gynecol. 2018 Jan;51(1):64–76.
Babu GR, Murthy GVS, Ana Y, Patel P, Deepa R, Neelon SEB, Kinra S, Reddy KS. Association of obesity with hypertension and type 2 diabetes mellitus in India: a meta-analysis of observational studies. World J Diabetes. 2018 Jan;9(1):40–52.
Jiang YQ, Tian Y, Zeng LJ, He SN, Zheng ZT, Shi L, Wang YJ, Wang YX, Yin XD, Liu XQ, Yang XC, Liu XP. The safety and efficacy of hybrid ablation for the treatment of atrial fibrillation: a meta-analysis. PLoS One. 2018 Jan 3;13(1):e0190170. DOI: http://dx.doi.org/10.1371/journal.pone.0190170.
Buczinski S, Gicquel E, Fecteau G, Takwoingi Y, Chigerwe M, Vandeweerd JM. Systematic review and meta-analysis of diagnostic accuracy of serum refractometry and brix refractometry for the diagnosis of inadequate transfer of passive immunity in calves. J Vet Intern Med. 2018 Jan;32(1):474–83.
Nakhjavan-Shahraki B, Yousefifard M, Rahimi-Movaghar V, Baikpour M, Nasirinezhad F, Safari S, Yaseri M, Moghadas Jafari A, Ghelichkhani P, Tafakhori A, Hosseini M. Transplantation of olfactory ensheathing cells on functional recovery and neuropathic pain after spinal cord injury: systematic review and meta-analysis. Sci Rep. 2018 Jan;8(1):325. DOI: http://dx.doi.org/10.1038/s41598-017-18754-4.
Chiarito M, Pagnesi M, Martino EA, Pighi M, Scotti A, Biondi-Zoccai G, Latib A, Landoni G, Mario CD, Margonato A, Maisano F, Feldman T, Alfieri O, Colombo A, Godino C. Outcome after percutaneous edge-to-edge mitral repair for functional and degenerative mitral regurgitation: a systematic review and meta-analysis. Heart. 2018 Feb;104(4):306–12.
Hummel P, Saxena A, Klingler C. Rapid qualitative review of ethical issues surrounding healthcare for pregnant women or women of reproductive age in epidemic outbreaks. Epidemiol Health. 2018 Jan 25;40:e2018003. DOI: http://dx.doi.org/10.4178/epih.e2018003.
Bos M, Nikkels PGJ, Cohen D, Schoones JW, Bloemenkamp KWM, Bruijn JA, Baelde HJ, van der Hoorn MLP, Turner RJ. Towards standardized criteria for diagnosing chronic intervillositis of unknown etiology: a systematic review. Placenta. 2018 Jan;61:80–8.
De Boer J, Prikken M, Lei WU, Begemann M, Sommer I. The effect of raloxifene augmentation in men and women with a schizophrenia spectrum disorder: a systematic review and meta-analysis. NPJ Schizophr. 2018 Jan;4(1):1. DOI: http://dx.doi.org/10.1038/s41537-017-0043-3.
De Groot AF, Appelman-Dijkstra NM, van der Burg SH, Kroep JR. The anti-tumor effect of RANKL inhibition in malignant solid tumors. a systematic review. Cancer Treat Rev. 2018 Jan;62:18–28.
Adam S, Feller A, Rohrmann S, Arndt V. Health-related quality of life among long-term (≥=5 years) prostate cancer survivors by primary intervention: a systematic review. Health Qual Life Outcomes. 2018 Jan;16(1):22. DOI: http://dx.doi.org/10.1186/s12955-017-0836-0.
Boatin AA, Cullinane F, Torloni MR, Betran AP. Audit and feedback using the Robson classification to reduce caesarean section rates: a systematic review. BJOG. 2018 Jan;125(1):36–42.
Chen H, Chai Y, Dong L, Niu W, Zhang P. Effectiveness and appropriateness of mHealth interventions for maternal and child health: systematic review. JMIR MHealth UHealth. 2018 Jan;6(1):e7. DOI: http://dx.doi.org/10.2196/mhealth.8998.
