Evaluating acute image ordering for real-world patient cases via language model alignment with radiological...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Imaging

Background:

• Diagnostic imaging is crucial for acute care but ordering is variable.
• Generative AI and large language models (LLMs) show promise for guideline-adherent imaging recommendations.
• Challenges remain in ensuring AI recommendations align with limited acute care diagnostic information.

Purpose of the Study:

• To introduce a framework leveraging LLMs for recommending imaging studies aligned with the American College of Radiology's Appropriateness Criteria.
• To enable state-of-the-art LLMs to reason about appropriate imaging choices for patient cases.

Main Methods:

• Developed a framework to intelligently leverage language models for imaging study recommendations.
• Created RadCases, a dataset of over 1500 annotated case summaries.
• Applied the framework to LLMs using the RadCases dataset.

Main Results:

• LLMs achieved accuracy comparable to clinicians in ordering imaging studies using the framework.
• The LLM pipeline functions as an intelligent assistant for clinicians.
• Improved accuracy of acute image ordering aligned with ACR Appropriateness Criteria.

Conclusions:

• Validated a strategy for AI software to enhance trustworthy clinical decision-making.
• AI-driven recommendations improve adherence to expert evidence-based guidelines.
• The framework supports improved diagnostic imaging practices in acute care settings.