Automating expert-level medical reasoning evaluation of large language models.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly used in clinical decision-making.
• Current LLM evaluation methods for medical reasoning lack rigor and scalability.
• A standardized benchmark for assessing medical reasoning in LLMs is needed.

Purpose of the Study:

• To introduce MedThink-Bench, a benchmark for rigorous and scalable assessment of LLMs' medical reasoning capabilities.
• To present LLM-w-Rationale, an evaluation framework for assessing LLM reasoning quality with expert-level fidelity and scalability.

Main Methods:

• Developed MedThink-Bench with 500 high-complexity medical questions across ten domains.
• Included expert-authored, step-by-step rationales for each question.
• Introduced LLM-w-Rationale, combining fine-grained rationale assessment with an LLM-as-a-Judge approach.

Main Results:

• LLM-w-Rationale demonstrated strong correlation with expert evaluations (Pearson coefficient up to 0.87).
• The LLM-w-Rationale framework achieved expert-level fidelity in evaluating reasoning quality.
• Evaluation time was reduced to only 1.4% of traditional expert evaluation time.

Conclusions:

• MedThink-Bench provides a rigorous and scalable standard for evaluating LLM medical reasoning.
• LLM-w-Rationale enables efficient and reliable assessment of LLM reasoning quality.
• These advancements support the safe and responsible deployment of LLMs in clinical practice.