Comparing AI chat tools and PubMed for medical literature searches

This investigation compared six AI chat tools (ChatGPT, Claude, Copilot, Gemini, OpenEvidence, and Perplexity) to each other and to PubMed to assess their utility in medical literature searches.

Each tool was prompted with two standardized prompts simulating an academic and a clinical literature need. Prompts were repeated three times per tool, and results recorded for each prompt iteration.

Main findings include the range in number of real and hallucinated articles for each AI tool, from no hallucinations in OpenEvidence to more than 80% hallucinated citations in ChatGPT. Across all tools, most real articles were assessed as relevant. Continued prompting of all tools produced duplicate results. Most tools showed very little consistency in citations provided to different researchers, suggesting low probability of replicating search results.

Though limited by small sample size, inconsistent data, and the rapid pace of change in LLM technology, this study suggests some advantages and drawbacks of using AI tools for literature searching compared with PubMed. AI tools with few or no hallucinations and high relevance, such as OpenEvidence and Perplexity, performed well in terms of simply supplying a few article citations on the topic, but small results sets and lack of consistency and transparency limit their value for thorough research. These results suggest that AI tools are not presently a viable replacement for PubMed or other academic databases for systematic reviews or other in-depth literature searches but may be adequate for limited information needs or as a complement to database or web searches.