Quantifying the reasoning abilities of LLMs on clinical cases

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large language models (LLMs) demonstrate advanced reasoning capabilities, yet their clinical application and reasoning evaluation in medicine are not well-established.
• Assessing the reasoning processes of LLMs in complex medical scenarios is crucial for safe and effective deployment.

Purpose of the Study:

• To introduce MedR-Bench, a novel benchmark for evaluating the medical reasoning of LLMs.
• To assess the performance of state-of-the-art reasoning LLMs across different stages of clinical care.

Main Methods:

• Developed MedR-Bench, comprising 1453 structured patient cases with reference reasoning across 13 body systems and 10 specialties.
• Created an evaluation framework covering examination recommendation, diagnostic decision-making, and treatment planning.
• Utilized the Reasoning Evaluator, an automated tool assessing reasoning efficiency, factual accuracy, and completeness.

Main Results:

• Current LLMs achieve over 85% accuracy in simple diagnostic tasks with sufficient data but falter in examination recommendation and treatment planning.
• LLM reasoning is generally factually accurate but frequently omits critical steps.
• Open-source LLMs are narrowing the performance gap with proprietary models.

Conclusions:

• While LLMs show potential in medical diagnosis, significant improvements are needed in higher-level clinical reasoning tasks.
• The development of robust benchmarks like MedR-Bench is essential for advancing AI in healthcare.
• Open-source advancements suggest a future of more accessible and equitable clinical AI tools.