How do search systems impact systematic searching? A qualitative study

Authors

Andy Hickner Education and Outreach Librarian, Weill Cornell Medicine, New York, NY https://orcid.org/0000-0003-1922-3813

DOI:

https://doi.org/10.5195/jmla.2023.1647

Keywords:

systematic reviews, user experience, thematic analysis, qualitative research, usability, expert searching, librarians, information retrieval, bibliographic databases, search engine

Abstract

Objective: Systematic reviews and other evidence synthesis projects require systematic search methods. Search systems require several essential attributes to support systematic searching; however, many systems used in evidence synthesis fail to meet one or more of these requirements. I undertook a qualitative study to examine the effects of these limitations on systematic searching and how searchers select information sources for evidence synthesis projects.

Methods: Qualitative data were collected from interviews with twelve systematic searchers. Data were analyzed using reflexive thematic analysis.

Results: I used thematic analysis to identify two key themes relating to search systems: systems shape search processes, and systematic searching occurs within the information market. Many systems required for systematic reviews, in particular sources of unpublished studies, are not designed for systematic searching. Participants described various workarounds for the limitations they encounter in these systems. Economic factors influence searchers’ selection of sources to search, as well as the degree to which vendors prioritize these users.

Conclusion: Interviews with systematic searchers suggest priorities for improving search systems, and barriers to improvement that must be overcome. Vendors must understand the unique requirements of systematic searching and recognize systematic searchers as a distinct group of users. Better interfaces and improved functionality will result in more efficient evidence synthesis.

References

Gusenbauer M, Haddaway NR. What every researcher should know about searching - clarified concepts, search advice, and an agenda to improve finding in academia. Res Synth Methods. 2021 Mar;12(2):136–47. DOI: https://doi.org/10.1002/jrsm.1457.

Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: A prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28–50. DOI: https://doi.org/10.18438%2Fb8dw3k.

Gann LB, Pratt GF. Using library search service metrics to demonstrate library value and manage workload. J Med Libr Assoc. 2013 Jul;101(3):227–9. DOI: https://doi.org/10.3163%2F1536-5050.101.3.015.

Bullers K, Howard AM, Hanson A, Kearns WD, Orriola JJ, Polo RL, Sakmar KA. It takes longer than you think: librarian time spent on systematic review tasks. J Med Libr Assoc. 2018 Apr 1;106(2):198–207. DOI: https://doi.org/10.5195/jmla.2018.323.

Bramer WM, de Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc. 2018 Oct;106(4):531–41. DOI: https://doi.org/10.5195%2Fjmla.2018.283.

Paynter R, Bañez LL, Berliner E, Erinoff E, Lege-Matsuura J, Potter S, Uhl S. EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. Rockville (MD): Agency for Healthcare Research and Quality (US); 2016.

Bethel A, Rogers M. A checklist to assess database-hosting platforms for designing and running searches for systematic reviews. Health Info Libr J. 2014 Mar;31(1):43–53. DOI: https://doi.org/10.1111/hir.12054.

Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods. 2020 Mar;11(2):181–217. DOI: https://doi.org/10.1002/jrsm.1378.

MacFarlane A, Russell-Rose T, Shokraneh F. Search Strategy Formulation for Systematic Reviews: issues, challenges and opportunities. Intelligent Systems with Applications. 2022 Jun;200091. DOI: https://doi.org/10.1016/j.iswa.2022.200091.

van Altena AJ, Spijker R, Olabarriaga SD. Usage of automation tools in systematic reviews. Res Synth Methods. 2019 Mar;10(1):72–82. DOI: https://doi.org/10.1002/jrsm.1335.

Cleo G, Scott AM, Islam F, Julien B, Beller E. Usability and acceptability of four systematic review automation software packages: a mixed method design. Syst Rev. 2019 Jun 20;8(1):145. DOI: https://doi.org/10.1186/s13643-019-1069-6.

Riaz M, Sulayman M, Salleh N, Mendes E. Experiences Conducting Systematic Reviews from Novices’ Perspective. BCS Learning & Development; 2010.

Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014 Sep;5(3):221–34. DOI: https://doi.org/10.1002/jrsm.1106.

Nielsen J. Interviewing Users. Nielsen Norman Group [Internet]. 2010 Jul 25 [cited 2022 Sep 21]; Available from: https://www.nngroup.com/articles/interviewing-users/.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006 Jan;3(2):77–101. DOI: https://psycnet.apa.org/doi/10.1191/1478088706qp063oa.

Braun V, Clarke V. Conceptual and design thinking for thematic analysis. Qualitative Psychology. 2021 May 13;9(1):3–26. DOI: https://psycnet.apa.org/doi/10.1037/qup0000196.

Braun V, Clarke V. Successful Qualitative Research: A Practical Guide for Beginners. 1st ed. Los Angeles: SAGE Publications Ltd; 2013.

Bunge M. Realism and antirealism in social science. Theory Decis. 1993 Nov;35(3):207–35. DOI: https://doi.org/10.1007/BF01075199.

Medical Library Association Public Services Caucus. Expertsearching email listserv [Internet]. [cited 2023 Feb 8]. Available from: https://pss.mlanet.org/mailman/listinfo/expertsearching_pss.mlanet.org.

University of Michigan Library. Systematic Reviews Workshop [Internet]. [cited 2023 Feb 8]. Available from: https://www.lib.umich.edu/research-and-scholarship/library-workshops-and-credit-courses/systematic-reviews-workshop.

Braun V, Clarke V. FAQs [Internet]. ThematicAnalysis.net. [cited 2022 Sep 21]. Available from: https://www.thematicanalysis.net/faqs.

Russell-Rose T, Shokraneh F. Designing the Structured Search Experience: Rethinking the Query-Builder Paradigm. Weave: Journal of Library User Experience. 2020 Mar 16;3(1). DOI: https://doi.org/10.3998/weave.12535642.0003.102.

Sperr E. PubVenn [Internet]. PubVenn. 2015 [cited 2022 Oct 10]. Available from: https://pubvenn.appspot.com/.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Info Libr J. 2015 Mar;32(1):5–22. DOI: https://doi.org/10.1111/hir.12083.

Hunter KE, Webster AC, Page MJ, Willson M, McDonald S, Berber S, Skeers P, Tan-Koay AG, Parkhill A, Seidler AL. Searching clinical trials registers: guide for systematic reviewers. BMJ. 2022 Apr 26;377:e068791. DOI: https://doi.org/10.1136/bmj-2021-068791.

Clark JM, Sanders S, Carter M, Honeyman D, Cleo G, Auld Y, Booth D, Condron P, Dalais C, Bateup S, Linthwaite B, May N, Munn J, Ramsay L, Rickett K, Rutter C, Smith A, Sondergeld P, Wallin M, Jones M, Beller E. Improving the translation of search strategies using the Polyglot Search Translator: a randomized controlled trial. J Med Libr Assoc. 2020 Apr 1;108(2):195–207. DOI: https://doi.org/10.5195%2Fjmla.2020.834.

Association of College & Research Libraries (ACRL). Framework for Information Literacy for Higher Education [Internet]. Association of College & Research Libraries (ACRL). 2016 [cited 2022 Aug 29]. Available from: https://www.ala.org/acrl/standards/ilframework.

Law EL-C, Roto V, Hassenzahl M, Vermeeren APOS, Kort J. Understanding, scoping and defining user experience: A survey approach. Proceedings of the 27th international conference on Human factors in computing systems - CHI 09. New York, New York, USA: ACM Press; 2009. p. 719.

García-Puente M, Pastor-Ramon E, Agirre O, Morán J-M, Herrera-Peco I. Research note. Open letter to the users of the new PubMed: a critical appraisal. EPI. 2020 Jun 23;29(3):e290336. DOI: https://doi.org/10.3145/epi.2020.may.36.

Banno M, Tsujimoto Y, Kataoka Y. Using the Cochrane Central Register of Controlled Trials to identify clinical trial registration is insufficient: a cross-sectional study. BMC Med Res Methodol. 2020 Jul 25;20(1):200. DOI: https://doi.org/10.1186/s12874-020-01083-y.