External Validation of Two Clinical Prediction Models for Pediatric Pneumonia
Sriram Ramgopal1, Melissa Neveu2, Douglas Lorenz3
1Division of Emergency Medicine (S Ramgopal, J Benedetti, J Lavey, and TA Florin), Ann & Robert H. Lurie Children's Hospital of Chicago, Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, Ill.
Insights
The Pneumonia Risk Score (PRS) model showed better accuracy in predicting pediatric pneumonia than the CARPE DIEM model. However, neither model is yet sufficient for independent clinical use in children.
Area of Science:
- Pediatric Emergency Medicine
- Diagnostic Accuracy
- Clinical Prediction Models
Background:
- Accurate diagnosis of pediatric pneumonia is crucial for effective treatment.
- Existing prediction models require external validation to assess their real-world performance.
Purpose of the Study:
- To externally validate the performance of the Pneumonia Risk Score (PRS) and CARPE DIEM models in predicting radiographic pneumonia in children.
- To assess the impact of recalibration and re-estimation on model performance.
Main Methods:
- Prospective evaluation of 202 children (90 days - 18 years) in a pediatric emergency department.
- Assessed performance of PRS and CARPE DIEM models using original and recalibrated/re-estimated parameters.
- Evaluated model performance via Area Under the Curve (AUC), sensitivity, and specificity.
Main Results:
- The PRS model demonstrated superior performance (AUC 0.72) compared to CARPE DIEM (AUC 0.59).
- PRS showed higher sensitivity (65.2%) and specificity (72.7%) than CARPE DIEM (sensitivity 56.5%, specificity 60.9%).
- Recalibration and re-estimation improved performance, especially for CARPE DIEM, enhancing calibration and diagnostic metrics.
Conclusions:
- The PRS model is more effective than CARPE DIEM for predicting pediatric radiographic pneumonia.
- Current models require further improvement and validation before independent clinical application.
- Clinical judgment remains essential for diagnosing pneumonia in children, even with prediction models.
Objective:
To externally validate two prediction models for pediatric radiographic pneumonia.
Methods:
We prospectively evaluated the performance of two prediction models (Pneumonia Risk Score [PRS] and Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine [CARPE DIEM] models) from a prospective convenience sample of children 90 days - 18 years of age from a pediatric emergency department undergoing chest radiography for suspected pneumonia between January 1, 2022, and December 31st, 2023. We evaluated model performance using the original intercepts and coefficients and evaluated for performance changes when performing recalibration and re-estimation procedures.
Results:
We included 202 patients (median age 3 years, IQR 1-6 years), of whom radiographic pneumonia was found in 92 (41.0%). The PRS model had an area under the receiver operator characteristic curve of 0.72 (95% confidence interval [CI] 0.64-0.79), which was higher than the CARPE DIEM (0.59; 95% CI 0.51-0.67) (P < 0.01). Using optimal cutpoints, the PRS model showed higher sensitivity (65.2%, 95% CI 54.6-74.9) and specificity (72.7%, 95% CI 63.4-80.8) compared to the CARPE DIEM model (sensitivity 56.5 [95% CI 45.8-66.8]; specificity 60.9 [95% CI 50.2-69.2]). Recalibration and re-estimation of models improved performance, particularly for the CARPE DIEM model, with gains in sensitivity and specificity, and improved calibration.
Conclusion:
The PRS model demonstrated better performance than the CARPE DIEM model in predicting radiographic pneumonia. Among children with a high rate of pneumonia, these models did not reach a level of performance sufficient to be used independently of clinical judgment. These findings highlight the need for further validation and improvement of models to enhance their utility.
Related Concept Videos
01:30Pneumonia III: Complications and Assessment
01:13Sensitivity, Specificity, and Predicted Value
Sensitivity is the...