Automated Chest X-ray Report Generation Remains Unsolved.

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Radiology Reporting

Background:

• Accurate chest radiograph interpretation is crucial for patient care but burdensome for radiologists.
• Existing AI models for automated report generation lack standardized evaluation frameworks.
• Limited benchmarks hinder the assessment of AI performance in clinical settings.

Purpose of the Study:

• To introduce the ReXrank Challenge V1.0 for evaluating AI models in chest radiograph report generation.
• To utilize ReXGradient, the largest test dataset (10,000 studies, 67 sites), for comprehensive benchmarking.
• To analyze model performance across normal/abnormal studies, generalization, and clinical finding identification.

Main Methods:

• Organized a competition (ReXrank Challenge V1.0) involving diverse participants and state-of-the-art AI models.
• Employed a large-scale dataset (ReXGradient) for training and testing.
• Utilized multiple metrics for comprehensive performance evaluation across various dimensions.

Main Results:

• Automated chest X-ray report generation is not yet a solved problem.
• Significant performance disparities exist between normal and abnormal study interpretations.
• Top models achieve less than 45% error-free reporting on abnormal cases, with notable variability across healthcare institutions.

Conclusions:

• Current AI systems for chest X-ray report generation require substantial improvement before clinical deployment.
• Addressing performance gaps in abnormal cases and ensuring cross-institutional generalization are critical research areas.
• Continued development is necessary to create robust and clinically reliable automated reporting systems.