Large Language Models lack essential metacognition for reliable medical reasoning

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Metacognition in AI

Background:

• Large Language Models (LLMs) exhibit expert-level performance on medical board examinations.
• The potential of LLMs for clinical decision support is significant.
• Metacognitive abilities, essential for medical reasoning, are largely unexplored in LLMs.

Purpose of the Study:

• To develop and introduce MetaMedQA, a novel benchmark for evaluating LLM metacognition in a medical context.
• To assess LLMs' ability to gauge their own knowledge and confidence.
• To identify metacognitive deficiencies in current LLMs relevant to clinical decision-making.

Main Methods:

• Created MetaMedQA, a benchmark dataset with confidence scores and metacognitive tasks integrated into medical multiple-choice questions.
• Evaluated twelve LLMs on metrics including confidence-based accuracy, missing answer recall, and unknown recall.
• Analyzed model performance concerning their awareness of knowledge limitations.

Main Results:

• All evaluated LLMs demonstrated significant metacognitive deficiencies despite high accuracy on standard multiple-choice questions.
• Models frequently failed to identify their knowledge limitations, providing confident incorrect answers.
• A critical disconnect exists between LLMs' perceived and actual medical reasoning capabilities.

Conclusions:

• Current LLMs possess critical metacognitive deficits that pose substantial risks for clinical applications.
• The findings highlight the inadequacy of existing evaluation methods for assessing LLM reliability in healthcare.
• Development of robust evaluation frameworks incorporating metacognitive abilities is crucial for safe LLM-enhanced clinical decision support.