Evaluating the Capabilities of Generative AI Tools in Understanding Medical Papers: Qualitative Study

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Informatics

Background:

• Medical literature comprehension is time-consuming for clinicians.
• Efficient tools are needed to help doctors process complex medical papers.
• Large language models (LLMs) offer potential solutions for medical information processing.

Purpose of the Study:

• To critically assess and compare the comprehension capabilities of various LLMs.
• To evaluate LLM accuracy and efficiency in understanding medical research papers.
• To utilize the STROBE checklist for a standardized evaluation framework.

Main Methods:

• Methodological research evaluating generative AI tools in medical paper comprehension.
• A benchmark pipeline processed 50 PubMed medical research papers.
• Six LLMs (GPT-3.5-Turbo, GPT-4, PaLM 2, Claude v1, Gemini Pro) were compared against expert benchmarks using 15 STROBE-derived questions.

Main Results:

• GPT-3.5-Turbo achieved the highest accuracy (66.9%), followed by GPT-4-1106 (65.6%) and PaLM 2 (62.1%).
• Statistically significant performance differences were observed among LLMs (P<.001).
• Newer LLM versions generally outperformed older ones, with notable performance variations across different paper sections.

Conclusions:

• This is the first study to use retrieval augmented generation to evaluate LLMs for medical paper comprehension.
• LLMs demonstrate potential to improve medical research efficiency and support evidence-based decision-making.
• Further research is required to address question format influence, potential biases, and the rapid evolution of LLM technology.