Intelligence without intuition: a mixed-methods pilot study on reasoning models in musculoskeletal physiotherapy for...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Musculoskeletal Pain Management

Background:

• Musculoskeletal pain, particularly low-back pain, is a prevalent and complex condition.
• Effective management necessitates integrating biopsychosocial factors within evidence-based clinical reasoning.
• The role of advanced AI, specifically reasoning large language models (LLMs), in supporting clinical decision-making requires thorough evaluation.

Purpose of the Study:

• To conduct a comprehensive human evaluation of reasoning LLMs for clinical reasoning tasks.
• To assess the validity and reliability of LLMs in supporting complex medical decision-making.
• To identify strengths and weaknesses of current LLMs in clinical reasoning applications.

Main Methods:

• Human evaluators assessed reasoning LLM outputs based on criteria including reliability, conceptual reasoning, completeness, correctness, relevance, and usefulness.
• A qualitative analysis was performed to identify specific areas of deviation from expert clinical reasoning.
• LLM performance was compared across different state-of-the-art models.

Main Results:

• State-of-the-art reasoning LLMs demonstrated sufficient test-retest reliability.
• Models were generally competent or proficient across key performance metrics with no significant differences between them.
• Qualitative analysis revealed limitations in logical coherence, patient-centeredness, empathy, and intuition, with intuition being a notable area for improvement.

Conclusions:

• Reasoning LLMs show potential as tools to aid clinical reasoning in musculoskeletal pain management.
• A multidimensional evaluation framework is crucial for assessing LLM performance in healthcare.
• Further development is needed to enhance LLM capabilities in patient-centered communication and intuitive reasoning for clinical applications.