Comparative performance of large language models in structuring head CT radiology reports: multi-institutional...

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Medical Reports
• Diagnostic Accuracy Assessment

Background:

• Structuring free-text radiology reports is crucial for data analysis and clinical decision-making.
• Large Language Models (LLMs) offer potential for automating this process, but their diagnostic performance needs evaluation.
• Prompting strategies can significantly influence LLM output accuracy.

Purpose of the Study:

• To compare the diagnostic performance of Claude, GPT, and Gemini LLMs in structuring Japanese head CT reports.
• To evaluate the impact of Standard, Chain of Thought, and Self-Consistency prompting on LLM accuracy for intracranial hemorrhage and skull fractures.
• To identify challenges and areas for improvement in LLM-based radiology report structuring.

Main Methods:

• Retrospective analysis of 3949 head CT reports from the Japan Medical Imaging Database (2018-2023).
• Ground truth established by two board-certified radiologists for intracranial hemorrhage and skull fractures.
• Three LLMs (Claude, GPT, Gemini) were tested with three prompting strategies; performance measured by accuracy, precision, recall, and F1-score.

Main Results:

• All nine LLM-prompt combinations demonstrated very high accuracy.
• Claude significantly outperformed GPT and Gemini in accuracy for both intracranial hemorrhage and skull fractures (p < 0.0001).
• Gemini's accuracy notably improved with Chain of Thought prompting; common errors involved ambiguous phrases and irrelevant findings.

Conclusions:

• Proprietary LLMs show strong potential for structuring free-text head CT reports for intracranial hemorrhage and skull fractures.
• Prompting techniques play a vital role in optimizing LLM diagnostic performance.
• Further research should focus on refining prompts and validating these AI tools in prospective, multilingual clinical settings.