Comparing Five Generative AI Chatbots' Answers to LLM-Generated Clinical Questions with Medical Information...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly used for clinical question answering.
• Evaluating LLM performance against human experts is crucial for safe integration into healthcare.

Purpose of the Study:

• To compare the accuracy and completeness of answers provided by five leading LLM chatbots to clinical questions against those of medical information scientists.
• To assess the alignment of LLM-generated answers with expert-curated information.

Main Methods:

• 5 LLM chatbots (ChatGPT, Gemini, Copilot, DeepSeek, Grok-3) were prompted with 45 PICO questions.
• Answers were compared to those generated by medical information scientists.
• Alignment was categorized as Total, Partial, or No Alignment by independent scientists.
• Partially aligned answers were analyzed for additional information provided.

Main Results:

• No significant performance differences were found between the five LLM chatbots in alignment ratings (p=0.46).
• LLMs achieved Total Alignment 20.9% of the time and Partial Alignment 78.7% of the time.
• Medical information scientists provided significantly more additional relevant information in partially aligned answers compared to LLMs (p=0.02).

Conclusions:

• Current leading LLM chatbots demonstrate comparable performance in answering clinical questions relative to medical information scientists.
• While LLMs provide substantial information, human experts offer greater depth in supplementary details.
• Future research should focus on validating the accuracy and relevance of additional information provided by both LLMs and human experts.