Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model

Area of Science:

• Health Informatics
• Clinical Decision Support
• Machine Learning in Healthcare

Background:

• Clinical decision support tools, such as chronic kidney disease (CKD) risk models, require ongoing performance monitoring after deployment.
• Ensuring the reliability of predictive models in real-world clinical settings is crucial for patient care.

Purpose of the Study:

• To evaluate the effectiveness of three distinct metrics in detecting performance degradation of a deployed CKD risk model.
• To assess the utility of standardized mean differences (SMDs), a membership model, and response distribution analysis for monitoring model performance.

Main Methods:

• A CKD risk model, initially developed on retrospective data, was silently deployed on new patient admissions.
• Three monitoring metrics (SMDs, membership model, response distribution analysis) were applied during the silent deployment phase.
• Prospective model performance was calculated using observed outcomes to assess the metrics' ability to detect changes.

Main Results:

• The deployed CKD model showed a significant performance decrease (AUROC 0.63) compared to retrospective data (AUROC 0.76).
• SMDs revealed significant differences in 88% of input variables between retrospective and deployment data (p <0.05).
• The membership model (AUROC 0.71) and response distribution analysis both effectively discriminated between retrospective and deployment settings (p <0.0001).

Conclusions:

• The evaluated metrics, including SMDs, membership models, and response distribution analysis, show promise for early detection of performance decline in deployed predictive models.
• These monitoring strategies can help ensure the continued accuracy and reliability of clinical decision support tools.