Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solving.

Area of Science:

• Artificial Intelligence in Medicine
• Cognitive Science
• Natural Language Processing

Background:

• Large Language Models (LLMs) demonstrate high accuracy on medical question-answering (QA) benchmarks.
• Debate exists regarding LLMs' capacity for flexible clinical reasoning and susceptibility to cognitive biases.

Purpose of the Study:

• To evaluate if advanced reasoning LLMs exhibit enhanced cognitive flexibility in clinical reasoning.
• To assess LLM performance on the medicine abstraction and reasoning corpus (mARC) benchmark, which tests for the Einstellung effect.

Main Methods:

• Assessed reasoning models from OpenAI, Grok, Gemini, Claude, and DeepSeek families.
• Utilized the medicine abstraction and reasoning corpus (mARC), an adversarial medical QA benchmark.
• Measured performance against physician accuracy and susceptibility to Einstellung-based reasoning traps.

Main Results:

• Stronger reasoning LLMs avoided Einstellung-based traps more effectively than weaker models.
• Top models achieved human-level performance on mARC, with Claude reaching 75% accuracy.
• On physician-challenging questions, top models answered 50-70% correctly with high confidence, suggesting reduced susceptibility to Einstellung effects compared to humans.

Conclusions:

• Advanced reasoning LLMs exhibit improved flexibility in medical reasoning.
• These models achieve human-level performance on the mARC benchmark.
• LLMs may be less prone to certain cognitive biases like the Einstellung effect than human physicians.