Related Experiment Video
Updated: Jul 17, 2025
An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
Pediatric prognostic models predicting inhospital child mortality in resource-limited settings: An external
Morris Ogero1,2, John Ndiritu1, Rachel Sarguta1
1Department of Mathematics University of Nairobi Nairobi Kenya.
Insights
External validation of prognostic models like RISC-Malawi revealed they underestimate mortality risk. These models require recalibration for reliable use in new clinical settings.
Area of Science:
- Clinical epidemiology
- Biostatistics
- Health outcomes research
Background:
- Prognostic models aid clinical decisions but often lack external validation.
- Uncertainty in reliability and generalizability hinders their widespread application.
- This study addresses the need for external validation of existing models.
Purpose of the Study:
- To externally validate four prognostic models, including the RISC-Malawi model.
- To assess the transferability and clinical utility of these models in a new population.
- To identify the need for model recalibration to improve generalizability.
Main Methods:
- Utilized data from the Clinical Information Network (CIN) for external validation.
- Included 163,329 eligible patients with in-hospital mortality as the primary outcome.
- Assessed model performance using discrimination (AUC) and calibration (slope, intercept) after data imputation.
Main Results:
- The RISC-Malawi model showed an AUC of 0.77, while Lowlavaar et al. models had an AUC of 0.75.
- All validated models demonstrated calibration issues, markedly underestimating mortality risk.
- Case-fatality rates were 8.7% for RISC-Malawi and 5.3% for Lowlavaar et al. models.
Conclusions:
- Externally validated models showed significant underestimation of mortality risk.
- Current models may not be reliable for use outside their original development contexts.
- Recalibration is essential to enhance the generalizability and clinical applicability of these prognostic models.
Background And Aims:
Prognostic models provide evidence-based predictions and estimates of future outcomes, facilitating decision-making, patient care, and research. A few of these models have been externally validated, leading to uncertain reliability and generalizability. This study aims to externally validate four models to assess their transferability and usefulness in clinical practice. The models include the respiratory index of severity in children (RISC)-Malawi model and three other models by Lowlavaar et al.
Methods:
The study used data from the Clinical Information Network (CIN) to validate the four models where the primary outcome was in-hospital mortality. 163,329 patients met eligibility criteria. Missing data were imputed, and the logistic function was used to compute predicted risk of in-hospital mortality. Models' discriminatory ability and calibration were determined using area under the curve (AUC), calibration slope, and intercept.
Results:
The RISC-Malawi model had 50,669 pneumonia patients who met the eligibility criteria, of which the case-fatality ratio was 4406 (8.7%). Its AUC was 0.77 (95% CI: 0.77-0.78), whereas the calibration slope was 1.04 (95% CI: 1.00 -1.06), and calibration intercept was 0.81 (95% CI: 0.77-0.84). Regarding the external validation of Lowlavaar et al. models, 10,782 eligible patients were included, with an in-hospital mortality rate of 5.3%. The primary model's AUC was 0.75 (95% CI: 0.72-0.77), the calibration slope was 0.78 (95% CI: 0.71-0.84), and the calibration intercept was 0.37 (95% CI: 0.28-0.46). All models markedly underestimated the risk of mortality.
Conclusion:
All externally validated models exhibited either underestimation or overestimation of the risk as judged from calibration statistics. Hence, applying these models with confidence in settings other than their original development context may not be advisable. Our findings strongly suggest the need for recalibrating these model to enhance their generalizability.
Related Concept Videos
01:20Comparing the Survival Analysis of Two or More Groups
01:20Actuarial Approach
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
01:03Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...
01:21Cancer Survival Analysis
01:15Assumptions of Survival Analysis
01:24Kaplan-Meier Approach