Large Language Models in Radiologic Numerical Tasks: A Thorough Evaluation and Error Analysis.

Area of Science:

• Medical Imaging and Artificial Intelligence
• Natural Language Processing in Healthcare

Background:

• Large language models (LLMs) show promise in processing clinical text.
• Evaluating LLM performance in specific medical domains like radiology is crucial.

Purpose of the Study:

• To assess the performance of various LLMs on radiology numerical extraction and judgment tasks.
• To conduct a detailed error analysis of LLM outputs in these tasks.

Main Methods:

• Six radiology tasks were defined: three extraction (T-score, CBD diameter, lung nodule size) and three judgment (PET hypermetabolism, osteoporosis, CBD dilation).
• LLMs evaluated included Llama 3.1 8b, DeepSeek R1 distilled Llama 8b, OpenAI o1-mini, and OpenAI GPT-5-mini, using data from MIMIC III and institutional databases.
• Manual review and error analysis were performed on all incorrect LLM outputs.

Main Results:

• For extraction tasks, non-RL models (o1-mini, GPT-5-mini) achieved >95% accuracy, while Llama showed variability (86%-98.7%).
• In judgment tasks, o1-mini and GPT-5-mini achieved accuracies of 91.7% and 99.0% respectively, with 100% accuracy in osteoporosis detection.
• No mathematical errors were found in o1-mini and GPT-5-mini outputs. Answer-only format negatively impacted Llama and DeepSeek distilled Llama performance.

Conclusions:

• Reinforcement learning (RL) reasoning LLMs exhibit consistent high performance and accuracy in radiology numerical tasks, with no mathematical errors.
• Non-RL models can also achieve acceptable performance, depending on the specific task complexity.