MeSH and text-word search strategies: precision, recall, and their implications for library instruction
DOI:
https://doi.org/10.5195/jmla.2022.1283Keywords:
Precision, Recall, MeSH, MeSH Instruction, Relevant Recall MethodAbstract
Objective: This study compared the recall and precision of MeSH-term versus text-word searching to better understand psychosocial MeSH terms and to provide guidance on whether to include both strategies in an information literacy session or how much time should be spent on teaching each search strategy.
Methods: Using the relevant recall method, a total of 3,162 resources were considered and evaluated to form a gold standard set of 1,521 relevant resources. We compared resources discussing psychosocial aspects of children and adolescents living with type 1 diabetes using two search strategies: text-word strategy versus MeSH-term strategy. The frequency of MeSH terms, the MeSH hierarchy, and elements of each search strategy were also examined.
Results: Using the 1,521 relevant articles, we found that the text-word search strategy had 54% recall, while the MeSH-term strategy had 75% recall. Also, the precision of the text-word strategy was 34.4%, while the precision of the MeSH-term strategy was 47.7%. Therefore, the MeSH-term search strategy yielded both greater recall and greater precision. The MeSH strategy was also more complicated in design and usage than the text-word strategy.
Conclusions: This study demonstrates the effectiveness of text-word and MeSH search strategies on precision and recall. The combination of text-word and MeSH strategies is recommended to achieve the most comprehensive results. These results support the idea that MeSH or a similar controlled vocabulary should be taught to experienced and knowledgeable students and practitioners who require a myriad of resources for their literature searches.
References
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn't. BMJ. 1996 Jan 13;312(7023):71–2. DOI: https://doi.org/10.1136/bmj.312.7023.71.
Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods. 2020 Oct;11(2):181–217. DOI: https://doi.org/10.1002/jrsm.1378.
Gehanno JF, Rollin L, Le Jean T, Louvel A, Darmoni S, Shaw W. Precision and recall of search strategies for identifying studies on return-to-work in Medline. J Occup Rehabil. 2009 Sep;19(3):223–30. DOI: https://doi.org/10.1007/s10926-009-9177-0.
Farrell A, Goosney J, Hutchens J. Evaluation of the effectiveness of course integrated library instruction in an undergraduate nursing program. J Can Health Libr Assoc. 2013 Dec;34(3):164–75. DOI: https://doi.org/10.5596/c13-061.
Purnell M, Royal B, Warton L. Supporting the development of information literacy skills and knowledge in undergraduate nursing students: an integrative review. Nurse Educ Today. 2020 Dec;95:1–10. DOI: https://doi.org/10.1016/j.nedt.2020.104585.
Schweikhard AJ, Hoberecht T, Peterson A, Randall K. The impact of library tutorials on the information literacy skills of occupational therapy and physical therapy students in an evidence-based practice course: a rubric assessment. Med Ref Serv Q. 2018 Jan-Mar;37(1):43–59. DOI: https://doi.org/10.1080/02763869.2018.1404388. PMID: 29327995.
Carlock D, Anderson J. Teaching and assessing the database searching skills of student nurses. Nurse Educ. 2007 Nov-Dec;32(6):251–5. DOI: https://doi.org/10.1097/01.NNE.0000299477.57185.ba. PMID: 17998852.
Jacobsen HE, Andenæs R. Third year nursing students' understanding of how to find and evaluate information from bibliographic databases and Internet sites. Nurse Educ Today. 2011 Nov;31(8):898–903. DOI: https://doi.org/10.1016/j.nedt.2011.01.003. Epub 2011 Feb 1. PMID: 21288607.
Diekema AR, Hopkins ES, Patterson B, Schvaneveldt N. using information practices of nurses to reform information literacy instruction in baccalaureate nursing programs. Evid Based Libr Inf Pract, 2019 Dec;14(4):72–102. DOI: https://doi.org/10.18438/eblip29588.
National Library of Medicine. Suggestions for finding author keywords using MeSH tools [Internet]. National Library of Medicine; 2020 [cited 8 July 2021]. <https://www.nlm.nih.gov/mesh/authors.html>.
National Library of Medicine. Learn about medical subject headings [Internet]. National Library of Medicine; 2020 [cited 8 July 2021]. <https://learn.nlm.nih.gov/documentation/training-packets/T000101112/>.
Willis S. NLM classification 2021 winter edition now available. NLM Tech Bull. 2021 Jan-Feb;(438):e3. <https://www.nlm.nih.gov/pubs/techbull/jf21/jf21_nlm_classification_winter_2021.html>.
Baumann N. How to use the medical subject headings (MeSH). Int J Clin Pract. 2016 Feb;70(2):171–4. DOI: 10.1111/ijcp.12767.
Van Altena AJ, Spijker R, Olabarriaga SD. Usage of automation tools in systematic reviews. Res Synth Methods. 2018 Dec;10(1):72–82. DOI: https://doi.org/10.1002/jrsm.1335.
King G, Lam P, Roberts ME. Computer-assisted keyword and document set discovery from unstructured text. Am J Pol Sci. 2017 Apr;61(4):971–88. DOI: https://doi.org/10.1111/ajps.12291.
Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015 Mar;10(3):e0118432. DOI: https://doi.org/10.1371/journal.pone.0118432.
Yao X, Van Durme B. Information extraction over structured data: question answering with Freebase [Internet]. Baltimore, MD: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics; 2014 June; 956–66. <https://www.aclweb.org/anthology/P14-1090.pdf>.
Sebastian Y, Siew E, & Orimaye SO. Predicting future links between disjoint research areas using heterogeneous bibliographic information network. Adv Knowl Discov Data Min. 2015 May;610–21. DOI: https://doi.org/10.1007/978-3-319-18032-8_48.
Thilakaratne M, Falkner K, Atapattu T. A systematic review on literature-based discovery workflow. PeerJ Comput Sci. 2019 Nov 18;5:e235. DOI: https://doi.org/10.7717/peerj-cs.235.
Linder F. Improved data collection from online sources using query expansion and active learning. SSRN Electronic J. 2017 Aug. DOI: https://doi.org/10.2139/ssrn.3026393.
Lowe HJ, Barnett GO. Understanding and using the medical subject headings (MeSH) vocabulary to perform literature searches. JAMA. 1994 Apr 13;271(14):1103–8. PMID: 8151853.
Chang AA, Heskett KM, Davidson TM. Searching the literature using medical subject headings versus text word with PubMed. Laryngoscope. 2006 Feb;116(2):336–40. DOI: https://doi.org/10.1097/01.mlg.0000195371.72887.a2.
Oregon Institute of Technology. Keyword vs. subject searching [Internet]. Oregon Institute of Technology; 2021 [cited 11 May 2021]. <https://www.oit.edu/library/help/library-guides/keyword-vs-subject>.
Columbia Universities Library. Keyword search [Internet]. Columbia Universities Library; 2021[cited 11 May 2021]. <http://www.columbia.edu/cu/lweb/help/clio/keyword.html>.
National Library of Medicine. MEDLINE, PubMed, and PMC (PubMed Central): how are they different? [Internet]. National Library of Medicine; 2020 [cited 11 May 2021]. <https://www.nlm.nih.gov/bsd/difference.html>.
Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994 Nov-Dec;1(6):447–58. DOI: https://doi.org/10.1136/jamia.1994.95153434.
Jenuwine ES, Floyd JA. Comparison of Medical Subject Headings and text-word searches in MEDLINE to retrieve studies on sleep in healthy individuals. J Med Libr Assoc. 2004 Jul;92(3):349–53.
Richter RR, Austin TM. Using MeSH (medical subject headings) to enhance PubMed search strategies for evidence-based practice in physical therapy. Phys Ther. 2012 Jan;92(1):124–32. DOI: https://doi.org/10.2522/ptj.20100178.
Gault LV, Shultz M, Davies KJ. Variations in Medical Subject Headings (MeSH) mapping: from the natural language of patron terms to the controlled vocabulary of mapped lists. J Med Libr Assoc. 2002 Apr;90(2):173–80.
Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007 Jun 15;7:16. DOI: https://doi.org/10.1186/1472-6947-7-16.
Damarell RA, May N, Hammond S, Sladek RM & Tieman JJ. Topic search filters: a systematic scoping review. Health Info Libr J 2019;36(1):4–40. DOI: https://doi.org/10.1111/hir.12244.
Liu YH, Wacholder N. Evaluating the impact of mesh (medical subject headings) terms on different types of searchers. Info Process Manag. 2017 July;53(4):851–70. DOI: https://doi.org/10.1016/j.ipm.2017.03.004.
Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28–50. DOI: https://doi.org/10.18438/B8DW3K.
Damarell RA, Lewis S, Trenerry C, Tieman JJ. Integrated care search: development and validation of a PubMed search filter for retrieving the integrated care research evidence. BMC Med Res Methodol. 2020 Jan;20(1):1–16. DOI: 10.1186/s12874-020-0901-y.
Nylander E, Ramstrand N, Hjort M, Rusaw DF. Development and validation of a sensitive MEDLINE search strategy to identify literature relevant to limb protheses. Prosthet Orthot Int. 2021 Jun;45(3):289–94. DOI: https://doi.org/10.1097/PXR.0000000000000019.
Hughes C, Filimonov M, Wray A, Spasić I. Leaving no stone unturned: flexible retrieval of idiomatic expressions from a large text corpus. Mach Learn Knowl Extr. 2021 Mar;3(1):263–83. DOI: https://doi.org/10.3390/make3010013.
Frické M. Measuring recall. J Inf Sci. 1998 Dec;24(6):409–17. DOI: https://doi.org/10.1177/016555159802400604.
Bell T, Hazel EG. Psychosocial evaluation for youth with type 1 diabetes: a 10-year review. Invited submission. Curr Diab Rep 2021 November.
National Library of Medicine. MeSH tree structures [Internet]. National Library of Medicine; 2018 [cited 3 May 2021]. <https://www.nlm.nih.gov/mesh/intro_trees.html>.
Ting KM. Precision and recall. In: Sammut C, Webb GI, eds. Encyclopedia of machine learning. Boston, MA: Springer Publishing Company; 2011. DOI: https://doi.org/10.1007/978-0-387-30164.
National Library of Medicine. MeSH changes and PubMed searching [Internet]. National Library of Medicine; 2018 [cited 3 May 2021]. <https://learn.nlm.nih.gov/documentation/training-packets/T0042010P/>.
National Library of Medicine. The MeSH database [Internet]. National Library of Medicine; 2018 [cited 3 May 2021]. <https://www.nlm.nih.gov/bsd/disted/meshtutorial/themeshdatabase/index.html>.
Additional Files
Published
Issue
Section
License
Copyright (c) 2022 Michelle M. DeMars, Carol Perruso
This work is licensed under a Creative Commons Attribution 4.0 International License.
