Comparing AI chat tools and PubMed for medical literature searches
DOI:
https://doi.org/10.5195/jmla.2026.2284Keywords:
AI, literature searches, comparisonAbstract
Objective: This investigation compared six AI chat tools (ChatGPT, Claude, Copilot, Gemini, OpenEvidence, and Perplexity) to each other and to PubMed to assess their utility in medical literature searches.
Methods: Each tool was prompted with two standardized prompts simulating an academic and a clinical literature need. Prompts were repeated three times per tool, and results recorded for each prompt iteration.
Results: Main findings include the range in number of real and hallucinated articles for each AI tool, from no hallucinations in OpenEvidence to more than 80% hallucinated citations in ChatGPT. Across all tools, most real articles were assessed as relevant. Continued prompting of all tools produced duplicate results. Most tools showed very little consistency in citations provided to different researchers, suggesting low probability of replicating search results.
Conclusions: Though limited by small sample size, inconsistent data, and the rapid pace of change in LLM technology, this study suggests some advantages and drawbacks of using AI tools for literature searching compared with PubMed. AI tools with few or no hallucinations and high relevance, such as OpenEvidence and Perplexity, performed well in terms of simply supplying a few article citations on the topic, but small results sets and lack of consistency and transparency limit their value for thorough research. These results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for systematic reviews or other in-depth literature searches but may be adequate for limited information needs or as a complement to database or web searches.
References
1. Gwon YN, Kim JH, Chung HS, Jung EJ, Chun J, Lee S, Shim SR. The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation. JMIR Med Inform. 2024 May 14;12:e51187.
2. Lund BD, Khan D, Yuvaraj M. ChatGPT in medical libraries, possibilities and future directions: An integrative review. Health Inf Libr J. 2024 Mar;41(1):4–15.
3. Vaishya R, Misra A, Vaish A. ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr. 2023 Apr;17(4):102744.
4. Sen S. AI and generative AI in health and medical libraries: a scoping review of present use and emerging potential. J EAHIL [Internet]. 2025 Jun 10 [cited 2025 Jul 21];21(2). Available from: https://ojs.eahil.eu/JEAHIL/article/view/675.
5. Wu J, Ma Y, Wang J, Xiao M. The Application of ChatGPT in Medicine: A Scoping Review and Bibliometric Analysis. J Multidiscip Healthc. 2024;17:1681–92.
6. Xu AY, Piranio VS, Speakman S, Rosen CD, Lu S, Lamprecht C, Medina RE, Corrielus M, Griffin IT, Chatham CE, Abchee NJ, Stribling D, Huynh PB, Harrell H, Shickel B, Brennan M. A Pilot Study of Medical Student Opinions on Large Language Models. Cureus. 2024 Oct;16(10):e71946.
7. Arrivé L, Minssen L, Ali A. ChatGPT risk of fabrication in literature searches. Comment on Br J Anaesth 2023; 131: e29-e30. Br J Anaesth. 2023 Nov;131(5):e172–3.
8. Blum M. ChatGPT Produces Fabricated References and Falsehoods When Used for Scientific Literature Search. J Card Fail. 2023 Sep;29(9):1332–4.
9. Chelli M, Descamps J, Lavoué V, Trojani C, Azar M, Deckert M, Raynier JL, Clowez G, Boileau P, Ruetsch-Chelli C. Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis. J Med Internet Res. 2024 May 22;26:e53164.
10. McGowan A, Gui Y, Dobbs M, Shuster S, Cotter M, Selloni A, Goodman M, Srivastava A, Cecchi GA, Corcoran CM. ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search. Psychiatry Res. 2023 Aug 1;326:115334.
11. Pallivathukal G, Kyaw Soe HH, Donald PM, Samson RS, Hj Ismail AR. ChatGPT for Academic Purposes: Survey Among Undergraduate Healthcare Students in Malaysia. Cureus. 16(1):e53032.
12. Suppadungsuk S, Thongprayoon C, Krisanapan P, Tangpanithandee S, Garcia Valencia O, Miao J, Mekraksakit P, Kashani K, Cheungpasitporn W. Examining the Validity of ChatGPT in Identifying Relevant Nephrology Literature: Findings and Implications. J Clin Med. 2023 Aug 25;12(17):5550.
13. Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Sci Rep. 2023 Sep 7;13(1):14045.
14. Wong J, Kriegler C, Shrivastava A, Duimering A, Le C. Utility of Chatbot Literature Search in Radiation Oncology. J Cancer Educ [Internet]. 2024 Dec 14 [cited 2025 Jul 21]; Available from: https://doi.org/10.1007/s13187-024-02547-1.
15. Zhu K, Zhang J, Klishin A, Esser M, Blumentals WA, Juhaeri J, Jouquelet-Royer C, Sinnott SJ. Evaluating the Accuracy of Responses by Large Language Models for Information on Disease Epidemiology. Pharmacoepidemiol Drug Saf. 2025 Feb;34(2):e70111.
16. Werf S van der. AI in literature research: a workshop perspective. J EAHIL [Internet]. 2025 Jun 10 [cited 2025 Jul 21];21(2). Available from: https://ojs.eahil.eu/JEAHIL/article/view/674.
17. Boyle A, Huo B, Sylla P, Calabrese E, Kumar S, Slater BJ, Walsh DS, Vosburg RW. Large language model-generated clinical practice guideline for appendicitis. Surg Endosc. 2025 Jun;39(6):3539–51.
18. Graf EM, McKinney JA, Dye AB, Lin L, Sanchez-Ramos L. Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles. Am J Perinatol. 2024 Nov;41(15):2072–81.
19. Jaźwińska K, Chandrasekar A. AI Search Has A Citation Problem [Internet]. Columbia Journalism Review. [cited 2025 Jul 21]. Available from: https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php
20. Roy RD, Gupta SD, Das D, Chowdhury PD. Chat GPT, Gemini or Meta AI: A comparison of AI platforms as a tool for answering higher-order questions in microbiology. J Postgrad Med. 2025 Jan 1;71(1):28–32.
21. Zhang JS, Yoon C, Williams DKA, Pinkas A. Exploring the Usage of ChatGPT Among Medical Students in the United States. J Med Educ Curric Dev. 2024 Jul 25;11:23821205241264695.
22. Daniel Nadler on OpenEvidence: Doctors’ Favorite AI App [Internet]. Sequoia Capital. [cited 2025 Oct 27]. Available from: https://sequoiacap.com/podcast/training-data-daniel-nadler/.
23. Hughes S, Bae M, Li M. Vectara Hallucination Leaderboard [Internet]. 2023 [cited 2025 Oct 12]. Available from: https://github.com/vectara/hallucination-leaderboard.
24. Xu Z, Jain S, Kankanhalli M. Hallucination is Inevitable: An Innate Limitation of Large Language Models [Internet]. arXiv; 2025 [cited 2025 Oct 12]. Available from: http://arxiv.org/abs/2401.11817.
25. Fiorini N, Canese K, Starchenko G, Kireev E, Kim W, Miller V, Osipov M, Kholodov M, Ismagilov R, Mohan S, Ostell J, Lu Z. Best Match: New relevance search for PubMed. PLoS Biol. 2018 Aug;16(8):e2005343.
26. Olson P. Ads ruined social media, now they’re coming to AI chatbots. The Economic Times of India [Internet]. 2025 Jun 2 [cited 2025 Jul 21]; Available from: https://economictimes.indiatimes.com/tech/catalysts/ads-ruined-social-media-now-theyre-coming-to-ai-chatbots-/articleshow/121567779.cms?from=mdr.
27. Brannon J. AI Advertising: Google AI Mode, ChatGPT & What’s Next | Coalition [Internet]. Coalition Technologies. 2025 [cited 2025 Oct 12]. Available from: https://coalitiontechnologies.com/blog/ai-advertising-chatgpt-google-ai-mode-ads.
28. Whitwam R. Google is quietly testing ads in AI chatbots [Internet]. Ars Technica. 2025 [cited 2025 Oct 12]. Available from: https://arstechnica.com/ai/2025/05/google-is-quietly-testing-ads-in-ai-chatbots/.
29. Rethlefsen ML, Brigham TJ, Price C, Moher D, Bouter LM, Kirkham JJ, Schroter S, Zeegers MP. Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study. J Clin Epidemiol. 2024 Feb;166:111229.
Published
Issue
Section
License
Copyright (c) 2026 A'Llyn Ettien, Kathleen Downing, Rebecca Drouhard, Bridget Gunn, Megan McNichol, Kate Silfen
This work is licensed under a Creative Commons Attribution 4.0 International License.
