Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Evidence-based medicine faces challenges with data availability and contextualization.
• Large language models (LLMs) offer potential solutions by summarizing literature and generating studies from real-world data.

Purpose of the Study:

• To evaluate the performance of different LLM systems in answering clinical questions.
• To compare general-purpose LLMs against specialized LLMs, including retrieval-augmented generation (RAG) and agentic systems.

Main Methods:

• Fifty clinical questions were posed to five LLM systems: OpenEvidence (RAG), ChatRWD (agentic), and three general-purpose LLMs (ChatGPT-4, Claude 3 Opus, Gemini 1.5 Pro).
• Nine physicians assessed the relevance, evidence quality, and actionability of the LLM-generated answers.

Main Results:

• General-purpose LLMs provided relevant, evidence-based answers for only 2-10% of questions.
• Specialized systems showed higher performance: OpenEvidence (RAG) at 24% and ChatRWD (agentic) at 58%.
• ChatRWD excelled in providing actionable answers for questions lacking existing literature (52%).

Conclusions:

• Specialized LLMs significantly outperform general-purpose LLMs for clinical question answering.
• Retrieval-augmented generation (RAG) systems are effective when existing data are available.
• Agentic LLMs, like ChatRWD, are vital for generating actionable insights in data-scarce scenarios.
• Future systems could combine RAG and agentic capabilities for comprehensive evidence support.