Limited generalizability of deep learning algorithm for pediatric pneumonia classification on external data.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Pediatric Radiology

Background:

• Deep learning systems (DLS) show promise for medical image analysis.
• Evaluating the generalizability of DLS models is crucial for clinical application.
• Pediatric pneumonia detection from chest radiographs is an area of active research.

Purpose of the Study:

• To develop a DLS for identifying pneumonia in pediatric chest radiographs.
• To assess the generalizability of the DLS by comparing performance on internal and external datasets.

Main Methods:

• A ResNet-50 deep convolutional neural network (DCNN) was trained on 5232 pediatric chest radiographs.
• The DCNN was tested on an internal dataset (624 radiographs) and an external dataset (383 radiographs).
• Performance was evaluated using receiver operating characteristic curves (AUC), and feature importance was visualized with class activation mapping (CAM).

Main Results:

• The DCNN achieved an AUC of 0.95 on the internal test set and 0.54 on the external test set (p < 0.0001).
• Class activation mapping (CAM) revealed the DCNN focused on relevant features for the internal set but not the external set.
• Significant performance disparity indicates issues with model generalizability.

Conclusions:

• The developed DLS demonstrated high performance on internal data but significantly lower accuracy on external data.
• Differences in feature relevance highlighted by heatmaps suggest a lack of generalizability.
• The study underscores the limitations of DLS generalizability in pediatric pneumonia detection and the need for robust validation.