Evaluating Reasoning Effect for LLMs: Prompt Sensitivity and Text-Image Based Performance in Musculoskeletal...

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology AI
• Machine Learning in Healthcare

Background:

• Multimodal large language models (LLMs) are emerging in radiology.
• The impact of LLM reasoning on text- and image-based tasks is not well understood.

Purpose of the Study:

• To evaluate the performance of reasoning-capable versus non-reasoning multimodal LLMs on musculoskeletal radiographic anatomy questions.
• To compare LLM performance against board-certified radiologists.

Main Methods:

• Four multimodal LLMs (two reasoning-capable, two non-reasoning) were tested.
• Tasks included 50 text-based and 50 arrow-localized MSK radiographic anatomy questions.
• Performance was benchmarked against two radiologists, with accuracy and error categorization.

Main Results:

• Reasoning LLMs significantly outperformed non-reasoning LLMs on text-based tasks (96% vs 94% accuracy).
• Reasoning LLMs showed superior performance on image-based tasks (70-72% vs 46-48% accuracy) but were less accurate than radiologists (88-90%).
• Common errors involved misidentification of adjacent structures and projection overlap.

Conclusions:

• LLM reasoning enhances performance in text-based radiology tasks and improves robustness.
• Multimodal LLMs currently have limitations in fine-grained visual understanding for image-based tasks.
• LLMs are best positioned as supportive tools in radiology, rather than autonomous diagnostic systems.