The Subgroup Imperative: Chest Radiograph Classifier Generalization Gaps in Patient, Setting, and Pathology Subgroups

Area of Science:

• Medical Imaging
• Artificial Intelligence in Healthcare
• Radiology Informatics

Background:

• Deep learning classifiers are increasingly used for chest radiograph interpretation.
• External validation on diverse, real-world datasets is crucial for assessing generalizability.
• Subgroup analysis is essential to identify performance disparities in AI algorithms.

Purpose of the Study:

• To externally evaluate the performance of four chest radiograph classifiers.
• To conduct a robust subgroup analysis based on patient, setting, and pathology factors.
• To assess the real-world applicability and potential biases of AI in radiology.

Main Methods:

• Retrospective analysis of 197,540 adult posteroanterior chest radiographs and reports (2016-2020).
• Ground truth labels generated using a validated open-source natural language processing tool.
• Performance metrics (accuracy, sensitivity, specificity, etc.) calculated for overall dataset and subgroups.

Main Results:

• Classifiers achieved 68%-77% accuracy, 64%-75% sensitivity, and 82%-94% specificity.
• Reduced sensitivity observed for solitary findings, younger patients (<40 years), and emergency department cases.
• Decreased specificity noted for normal radiographs with support devices; sex and ancestry impacted performance.

Conclusions:

• Chest radiograph classifier performance is influenced by patient, setting, and pathology variables.
• Subgroup analysis is critical for informing AI implementation and ensuring equitable performance.
• Ongoing monitoring is necessary to maintain quality, safety, and fairness in AI-assisted radiology.