Collaboration between clinicians and vision-language models in radiology report generation

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Radiology

Background:

• Automated radiology report generation can enhance patient care and reduce radiologist workload.
• Evaluating the clinical quality of AI-generated reports is a key challenge for real-world adoption.

Purpose of the Study:

• To build and evaluate a state-of-the-art AI system (Flamingo-CXR) for chest radiograph report generation.
• To assess the clinical quality and utility of AI-generated radiology reports through expert evaluation.

Main Methods:

• Developed Flamingo-CXR, an AI system for chest radiograph report generation.
• Engaged a panel of board-certified radiologists to evaluate AI-generated reports against clinician reports.
• Analyzed report preferences across different clinical settings and identified error rates in both AI and human reports.

Main Results:

• 56.1% of intensive care unit reports and 77.7% of overall in/outpatient reports were preferred or equivalent to clinician reports.
• 94% of reports for cases with no abnormal findings were preferred or equivalent.
• Significant error rates were found in both AI and human reports (24.8% shared errors).

Conclusions:

• Flamingo-CXR demonstrates significant potential for clinical utility in radiology report generation.
• Expert evaluation indicates AI reports are competitive with human reports, especially in specific settings.
• An assistive clinician-AI collaboration setting shows promise for addressing errors and enhancing report composition.