An evidence-based method for assessing the value of a search tool: a pilot study

Authors

  • Donald Stanley Pearson Manager, Patient and Family Resource Center, James Cancer Hospital and Solove Research Institute, The Ohio State University Comprehensive Cancer Center, Room C001, Ground Floor Lobby, 460 West 10th Avenue, Columbus, OH 43210 https://orcid.org/0000-0001-5488-8277
  • Stevo Roksandic Regional Director, Library Services, Mount Carmel Health System, 127 South Davis Avenue, Columbus, OH 43222
  • Jill Kilanowski Associate Dean, Graduate Program, and Professor, Mount Carmel College of Nursing, Columbus, OH

DOI:

https://doi.org/10.5195/jmla.2018.287

Keywords:

Libraries, Medical, Federated Search, Searching, User Behavior, Resident Physicians, Nursing Students, Graduate Students, Nursing Graduate Students, Pilot Study

Abstract

Objective: The objective of this study was to develop an evidence-based method with a set of metrics that could be used to assess an information search tool.

Methods: This pilot study analyzed a two-group convenience sample of graduate nursing students and resident physicians. The intervention group received ten minutes of instruction on a familiar search tool (eSearcher). Each group was provided one prompt to search for clinical guidelines on a given topic within their scope of practice and asked to find the best result using only eSearcher (intervention group) or specifically excluding eSearcher (comparison group). Three measurements of search results were employed: time elapsed to complete the search, an accuracy score, and a participant-reported score of confidence in the result.

Results: Forty-two students participated in this study (23 graduate nursing students and 19 resident physicians). The intervention group consisted of 22 participants (12 graduate nursing students and 10 resident physicians), and the comparison group consisted of 20 participants (11 graduate nursing students and 9 resident physicians). The intervention group had lower mean ranks in both accuracy and confidence compared to the comparison (not eSearcher) group, although these differences were not significant. However, the intervention (eSearcher) group had significantly longer search times compared to the comparison (not eSearcher) group.

Discussion: These findings provided new insights into the performance of the search tool and how users felt about their search experience. The quantitative evidence gained from this study led directly to an informed decision to explore other options for search tools. The evidence-based methods and process developed in this pilot study will enable similar studies to test other student groups and other search tools, leading to better informed purchasing and instructional decisions.

Author Biographies

Donald Stanley Pearson, Manager, Patient and Family Resource Center, James Cancer Hospital and Solove Research Institute, The Ohio State University Comprehensive Cancer Center, Room C001, Ground Floor Lobby, 460 West 10th Avenue, Columbus, OH 43210

Donald Pearson, MBA, MLIS, AHIP
Manager – Patient and Family Resource Center
The Ohio State University Comprehensive Cancer Center – James Cancer Hospital and Solove Research Institute
Room C001, Ground Floor Lobby, 460 West 10th Avenue
Columbus, OH  43210

614-366-8220

donald.pearson@osumc.edu

Creating a cancer-free world. One person, one discovery at a time.

 

Stevo Roksandic, Regional Director, Library Services, Mount Carmel Health System, 127 South Davis Avenue, Columbus, OH 43222

Stevo Roksandic, MBA, MLIS, AHIP
Regional Director
Mount Carmel Health System
Library Services
127 S. Davis Avenue
Columbus, OH 43222
614-234-1644 phone

Jill Kilanowski, Associate Dean, Graduate Program, and Professor, Mount Carmel College of Nursing, Columbus, OH

PhD, RN, APRN, CPNP, FAAN
Associate Dean Graduate Program, Professor

References

Breeding M. Library technology guides: discovery products [Internet]. Library Technology Guides [cited 11 Apr 2018]. <https://librarytechnology.org/discovery/>.

American Academy of Family Physicians. Clinical practice guideline manual [Internet]. The Academy [cited 6 Apr 2018]. <https://www.aafp.org/patient-care/clinical-recommendations/cpg-manual.html>.

Georgas H. Google vs. the library: student preferences and perceptions when doing research using Google and a federated search tool. portal: Libr Acad. 2013 Apr;13(2):165–85.

Belliston CJ, Howland JL, Roberts BC. Undergraduate use of federated searching: a survey of preferences and perceptions of value-added functionality. Coll Res Libr. 2007 Nov;68(6):472–87.

Thiele RH, Poiro NC, Scalzo DC, Nemergut EC. Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: results of a randomised trial. Postgrad Med J. 2010 Aug;86(1018):459–65. DOI: http://dx.doi.org/10.1136/pgmj.2010.098053.

Fagan JC, Mandernach MA, Nelson CS, Paulo JR, Saunders G. Usability test results for a discovery tool in an academic library. Inform Technol Libr. 2012 Mar;31(1):83–112.

Comeaux DJ. Usability testing of a web-scale discovery system at an academic library. Coll Undergrad Libr. 2012 Apr;19(2–4):189–206. DOI: http://dx.doi.org/10.1080/10691316.2012.695671.

Krause R, Moscati R, Halpern S, Schwartz DG, Abbas J. Can emergency medicine residents reliably use the Internet to answer clinical questions? West J Emerg Med. 2011 Nov;12(4):442–7. DOI: http://dx.doi.org/10.5811%2Fwestjem.2010.9.1895.

Downloads

Published

2018-10-04

Issue

Section

Original Investigation