Comparison of AI-generated radiology impressions: a multi-stakeholder evaluation

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Radiology Reporting

Background:

• Artificial intelligence (AI) is increasingly used in medical imaging.
• Evaluating the performance of AI-generated radiology reports is crucial for clinical adoption.
• Domain-specific AI models may offer advantages over general-purpose models.

Purpose of the Study:

• To compare the quality of AI-generated impressions from a custom domain-specific AI model and a general large language model (LLM) against original radiologist-authored impressions.
• To assess clinician preferences and perceptions of completeness, correctness, conciseness, clarity, clinical utility, and patient harm for different impression types.

Main Methods:

• Retrospective, blinded evaluation of 200 oncologic computed tomography (CT) reports.
• Comparison of original radiologist impressions, custom AI model impressions, and general LLM impressions.
• Rating of impressions by 10 clinicians (radiologists and oncologists) on key quality metrics.

Main Results:

• Radiologists preferred original and custom AI impressions over general LLM impressions.
• Oncologists showed no significant preference among impression types.
• Custom AI impressions demonstrated near parity with human-generated impressions in quality.
• General LLM impressions were longer and less concise but slightly more complete.
• Patient harm ratings were uniformly low across all impression types.

Conclusions:

• Domain-specific AI models show promise in generating high-quality radiology impressions.
• AI-generated impressions are perceived as safe and clinically useful by clinicians.
• Further research may optimize general LLMs for radiology reporting to improve conciseness and preference.