Clinical Validation of a Generative Artificial Intelligence Model for Chest Radiograph Reporting: A Multicohort Study

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Artificial intelligence (AI) is increasingly used for generating radiology reports, necessitating thorough clinical evaluation.
• The accuracy and acceptability of AI-generated reports require assessment against human-generated reports.

Purpose of the Study:

• To assess the clinical acceptability of AI-generated chest radiograph reports.
• To evaluate the accuracy of AI algorithms in identifying referable abnormalities in chest radiographs.

Main Methods:

• Retrospective collection of chest radiographs from diverse clinical settings (ICU, ED, health checkups, public dataset).
• Application of an automated AI algorithm for report generation.
• Evaluation of report acceptability by seven thoracic radiologists using standard and stringent criteria.
• Comparison of AI-generated and radiologist-written reports for acceptability, sensitivity, and specificity.

Main Results:

• AI and radiologist reports showed comparable acceptability under a standard criterion (88.4% vs 89.2%, P = .36).
• AI reports were less acceptable under a stringent criterion (66.8% vs 75.7%, P < .001), with a substantial need for revisions.
• AI reports demonstrated higher sensitivity (81.2% vs 59.4%, P < .001) but lower specificity (81.0% vs 93.6%, P < .001) for referable abnormalities.

Conclusions:

• AI-generated chest radiograph reports exhibit comparable acceptability to radiologist-written reports, though revisions are often needed.
• AI shows promise in improving sensitivity for abnormality detection but requires further refinement to enhance specificity.
• Radiologists surveyed expressed that current AI reports are not yet sufficient to replace human-generated reports.