Search methods for prognostic factor systematic reviews: a methodologic investigation
DOI:
https://doi.org/10.5195/jmla.2021.939Keywords:
Systematic Review Methods, Search Methods, Prognosis Studies as Topic, Prognosis FiltersAbstract
Objective: This study retroactively investigated the search methods used in the 2019 Hayden et al. study, one of the first systematic reviews of prognostic factors that was published in the Cochrane Library. The review was designed to address recognized weaknesses in reviews of prognosis by using multiple supplementary search methods in addition to traditional electronic database searching.
Methods: The authors used four approaches to comprehensively assess aspects of systematic review literature searching for prognostic factor studies: (1) comparison of search recall of broad versus focused electronic search strategies, (2) linking of search methods of origin for eligible studies, (3) analysis of impact of supplementary search methods on meta-analysis conclusions, and (4) analysis of prognosis filter performance.
Results: The review’s focused electronic search strategy resulted in a 91% reduction in recall, compared to a broader version. Had the team relied on the focused search strategy without using supplementary search methods, they would have missed 23 of 58 eligible studies that were indexed in MEDLINE; additionally, the number of included studies in 2 of the review’s primary outcome meta-analyses would have changed. Using a broader strategy without supplementary searches would still have missed 5 studies. The prognosis filter used in the review demonstrated the highest sensitivity of any of the filters tested.
Conclusions: Our study results support recommendations for supplementary search methods made by prominent systematic review methodologists. Leaving out any supplemental search methods would have resulted in missed studies, and these omissions would not have been prevented by using a broader search strategy or any of the other prognosis filters tested.References
Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, Udumyan R, Moons KGM, Steyerberg EW, Roberts I, Schroter S, Altman DG, Riley RD; PROGRESS Group. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013 Feb 5;346:e5595-e. DOI: http://dx.doi.org/10.1136/bmj.e5595.
Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, Hemingway H; PROGRESS Group. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med. 2013;10(2):e1001380. DOI: http://dx.doi.org/10.1371/journal.pmed.1001380.
Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG; PROGRESS Group. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381. DOI: http://dx.doi.org/10.1371/journal.pmed.1001381.
Hingorani AD, van der Windt DA, Riley RD, Abrams K, Moons KGM, Steyerberg EW, Schroter S, Sauerbrei W, Altman DG, Hemingway H; PROGRESS Group. Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ. 2013 Feb 5;346:e5793-e. DOI: http://dx.doi.org/10.1136/bmj.e5793.
Corp N, Jordan JL, Hayden JA, Irvin E, Parker R, Smith A, van der Windt DA. Protocol: a systematic review of studies developing and/or evaluating search strategies to identify prognosis studies. Syst Rev. 2017 Apr 20;6(1):88. DOI: http://dx.doi.org/10.1186/s13643-017-0482-y.
Yale University. Finding evidence in PubMed [Internet]. Newhaven, CT: Yale University; 2012 [updated 16 Jan 2015; cited 13 Jan 2020]. <https://library.medicine.yale.edu/tutorials/577>.
Walker-Dilks C, Wilczynski NL, Haynes RB; Hedges Team. Cumulative Index to Nursing and Allied Health Literature search strategies for identifying methodologically sound causation and prognosis studies. Appl Nurs Res. 2008 May;21(2):98–103. DOI: http://dx.doi.org/10.1016/j.apnr.2006.04.004.
Wilczynski NL, Haynes RB. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. J Am Med Inform Assoc. 2005 Jul;12(4):481–5. DOI: http://dx.doi.org/10.1197/jamia.M1752.
Wilczynski NL, Haynes RB; Hedges Team. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004 Jun 9;2:23. DOI: http://dx.doi.org/10.1186/1741-7015-2-23.
Chatterley T, Dennett L. Utilisation of search filters in systematic reviews of prognosis questions. Health Inf Libr J. 2012 Dec;29(4):309–22. DOI: http://dx.doi.org/10.1111/hir.12004.
Riley RD, Moons KGM, Snell KIE, Ensor J, Hooft L, Altman DG, Hayden J, Collins GS, Debray TPA. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019 Jan 30;364:k4597-k. DOI: http://dx.doi.org/10.1136/bmj.k4597.
Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons KGM. Search filters for finding prognostic and diagnostic prediction studies in MEDLINE to enhance systematic reviews. PLoS One. 2012;7(2):e32844. DOI: http://dx.doi.org/10.1371/journal.pone.0032844.
Holland JL, Wilczynski NL, Haynes RB; Hedges Team. Optimal search strategies for identifying sound clinical prediction studies in EMBASE. BMC Med Inform Decis Mak. 2005 Apr 29;5:11. DOI: http://dx.doi.org/10.1186/1472-6947-5-11.
Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001 Jul;8(4):391–7. DOI: http://dx.doi.org/10.1136/jamia.2001.0080391.
Keogh C, Wallace E, O’Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a web-based register. J Clin Epidemiol. 2011 Aug;64(8):848–60. DOI: http://dx.doi.org/10.1016/j.jclinepi.2010.11.011.
Wong SSL, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges T. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc. 2003:728–32.
Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064–5. DOI: http://dx.doi.org/10.1136/bmj.38636.593461.68.
Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results—guidance for future prognosis reviews. J Clin Epidemiol. 2009 Aug;62(8):781–96.e1. DOI: http://dx.doi.org/10.1016/j.jclinepi.2008.09.004.
Hayden JA, Tougas ME, Riley R, Iles R, Pincus T. Individual recovery expectations and prognosis of outcomes in non‐specific low back pain: prognostic factor exemplar review. Cochrane Database Syst Rev. 2014(9). DOI: http://dx.doi.org/10.1002/14651858.CD011284.
Fadyl J, McPherson K. Return to work after injury: a review of evidence regarding expectations and injury perceptions, and their influence on outcome. J Occup Rehabil. 2008 Dec;18(4):362–74. DOI: http://dx.doi.org/10.1007/s10926-008-9153-0.
Hallegraeff JM, Krijnen WP, van der Schans CP, de Greef MHG. Expectations about recovery from acute non-specific low back pain predict absence from usual work due to chronic low back pain: a systematic review. J Physiother. 2012 Sep;58(3):165–72. DOI: http://dx.doi.org/10.1016/S1836-9553(12)70107-8.
Iles RA, Davidson M, Taylor NF. Psychosocial predictors of failure to return to work in non-chronic non-specific low back pain: a systematic review. Occup Environ Med. 2008 Aug;65(8):507–17. DOI: http://dx.doi.org/10.1136/oem.2007.036046.
Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It’s in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health (Oxf). 2005 Dec;27(4):388–91. DOI: http://dx.doi.org/10.1093/pubmed/fdi056.
Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007 Apr 18(2):MR000001. DOI: http://dx.doi.org/10.1002/14651858.MR000001.pub2.
Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015 Apr;68(4):412–7. DOI: http://dx.doi.org/10.1016/j.jclinepi.2014.10.008.
Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement. Res Synth Methods. 2018 Jun;9(2):195–223. DOI: http://dx.doi.org/10.1002/jrsm.1286.
Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf MI, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS; Cochrane Information Retrieval Methods Group. Chapter 4: Searching for and selecting studies. In: Cochrane handbook for systematic reviews of interventions [Internet]. Version 6.0. Cochrane; Jul 2019 [cited 13 Jan 2020]. <https://training.cochrane.org/handbook/current/chapter-04>.
Hayden JA, Wilson MN, Riley RD, Iles R, Pincus T, Ogilvie R. Individual recovery expectations and prognosis of outcomes in non-specific low back pain: prognostic factor review. Cochrane Database Syst Rev. 2019 Nov 25;2019(11):CD011284. DOI: http://dx.doi.org/10.1002/14651858.CD011284.pub2.
Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis. Presented at: 14th Cochrane Colloquium; Dublin, Ireland; 23–26 Oct 2006.
Parker R, Tougas ME, Hayden JA. Validating prognosis search filters using relative recall based on prognosis systematic reviews. Poster presented at: 21st Cochrane Colloquium; Quebec City, QC, Canada; 19–23 Sep 2013.
Kok R, Verbeek JAHM, Faber B, van Dijk FJH, Hoving JL. A search strategy to identify studies on the prognosis of work disability: a diagnostic test framework. BMJ Open. 2015 May 19;5(5):e006315-e. DOI: http://dx.doi.org/10.1136/bmjopen-2014-006315.
Gehanno JF, Rollin L, Le Jean T, Louvel A, Darmoni S, Shaw W. Precision and recall of search strategies for identifying studies on return-to-work in MEDLINE. J Occup Rehabil. 2009 Sep;19(3):223–30. DOI: http://dx.doi.org/10.1007/s10926-009-9177-0.
Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Inf Libr J. 2008 Dec;25(4):313–7. DOI: http://dx.doi.org/10.1111/j.1471-1842.2008.00825.x.
Additional Files
- Appendix A: Focused Ovid MEDLINE search strategy used in Hayden et al.’s Cochrane review on the association between recovery expectations and disability outcomes in adults with low back pain
- Appendix B: Supplemental tables
- Appendix C: Data used in meta-analyses of association of expectations with outcomes
- Appendix D: Appendix D: Associated data
