MedCalc-Bench: Evaluating Large Language Models for Medical Calculations.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Clinical Decision Support

Background:

• Current benchmarks for evaluating large language models (LLMs) in medicine primarily assess domain knowledge and descriptive reasoning, not quantitative skills.
• Physicians frequently rely on clinical calculators employing quantitative equations and rule-based reasoning for evidence-based decision support.
• There is a need to evaluate the computational and logic-based reasoning capabilities of LLMs in medical contexts.

Purpose of the Study:

• To introduce MedCalc-Bench, a novel dataset designed to evaluate the medical calculation capabilities of LLMs.
• To assess the performance of current LLMs on quantitative medical reasoning tasks.
• To identify specific weaknesses in LLMs related to clinical calculations.

Main Methods:

• Development of MedCalc-Bench, a dataset comprising over 1000 manually reviewed instances from 55 distinct medical calculation tasks.
• Each instance includes a patient note, a question requiring a specific medical value computation, a ground truth answer, and a step-by-step explanation.
• Evaluation of existing LLMs using the MedCalc-Bench dataset.

Main Results:

• LLMs demonstrate potential in medical calculations but are not yet clinically viable.
• Common errors include incorrect entity extraction, misuse of equations or rules, and arithmetic inaccuracies.
• Significant gaps exist in LLMs' quantitative knowledge and reasoning abilities for clinical settings.

Conclusions:

• MedCalc-Bench serves as a crucial resource for benchmarking LLM performance in medical calculations.
• Current LLMs require substantial improvement to reliably perform clinical calculations.
• Future research should focus on enhancing LLMs' quantitative reasoning for diverse clinical applications.