Related Experiment Video
Updated: Oct 16, 2025
Asthma Detection Research Based on Voice Signal Processing and Machine Learning
Published on: July 22, 2025
Limited generalizability of deep learning algorithm for pediatric pneumonia classification on external data
Kevin Z Xin1, David Li2,3, Paul H Yi4,5
1Transitional Year Program, Mount Carmel Health System, Grove City, OH, USA.
Insights
A deep learning system (DLS) for pediatric pneumonia detection performed well internally but poorly on external data. This highlights challenges in generalizability for AI diagnostic tools in medical imaging.
Area of Science:
- Medical Imaging
- Artificial Intelligence
- Pediatric Radiology
Background:
- Deep learning systems (DLS) show promise for medical image analysis.
- Evaluating the generalizability of DLS models is crucial for clinical application.
- Pediatric pneumonia detection from chest radiographs is an area of active research.
Purpose of the Study:
- To develop a DLS for identifying pneumonia in pediatric chest radiographs.
- To assess the generalizability of the DLS by comparing performance on internal and external datasets.
Main Methods:
- A ResNet-50 deep convolutional neural network (DCNN) was trained on 5232 pediatric chest radiographs.
- The DCNN was tested on an internal dataset (624 radiographs) and an external dataset (383 radiographs).
- Performance was evaluated using receiver operating characteristic curves (AUC), and feature importance was visualized with class activation mapping (CAM).
Main Results:
- The DCNN achieved an AUC of 0.95 on the internal test set and 0.54 on the external test set (p < 0.0001).
- Class activation mapping (CAM) revealed the DCNN focused on relevant features for the internal set but not the external set.
- Significant performance disparity indicates issues with model generalizability.
Conclusions:
- The developed DLS demonstrated high performance on internal data but significantly lower accuracy on external data.
- Differences in feature relevance highlighted by heatmaps suggest a lack of generalizability.
- The study underscores the limitations of DLS generalizability in pediatric pneumonia detection and the need for robust validation.
Purpose:
(1) Develop a deep learning system (DLS) to identify pneumonia in pediatric chest radiographs, and (2) evaluate its generalizability by comparing its performance on internal versus external test datasets.
Methods:
Radiographs of patients between 1 and 5 years old from the Guangzhou Women and Children's Medical Center (Guangzhou dataset) and NIH ChestXray14 dataset were included. We utilized 5232 radiographs from the Guangzhou dataset to train a ResNet-50 deep convolutional neural network (DCNN) to identify pediatric pneumonia. DCNN testing was performed on a holdout set of 624 radiographs from the Guangzhou dataset (internal test set) and 383 radiographs from the NIH ChestXray14 dataset (external test set). Receiver operating characteristic curves were generated, and area under the curve (AUC) was compared via DeLong parametric method. Colored heatmaps were generated using class activation mapping (CAM) to identify important image pixels for DCNN decision-making.
Results:
The DCNN achieved AUC of 0.95 and 0.54 for identifying pneumonia on internal and external test sets, respectively (p < 0.0001). Heatmaps generated by the DCNN showed the algorithm focused on clinically relevant features for images from the internal test set, but not for images from the external test set.
Conclusion:
Our model had high performance when tested on an internal dataset but significantly lower accuracy when tested on an external dataset. Likewise, marked differences existed in the clinical relevance of features highlighted by heatmaps generated from internal versus external datasets. This study underscores potential limitations in the generalizability of such DLS models.
