Identifying unreliable predictions in clinical risk models.

Area of Science:

• Clinical decision support
• Biostatistics
• Health informatics

Background:

• Predictive risk stratification models are crucial for clinical decision-making.
• Standard performance metrics (e.g., accuracy) average performance across diverse patient groups.
• Assessing the reliability of individual predictions remains challenging.

Purpose of the Study:

• Introduce a novel method to identify patient subgroups where predictive models are likely to perform poorly.
• Develop an "unreliability score" to flag misleading predictions for individual patients.
• Address the challenge of model reliability in healthcare settings with large class imbalance.

Main Methods:

• Developed a new "unreliability score" applicable to any clinical risk model.
• Validated the method using data from over 40,000 patients in the Global Registry of Acute Coronary Events (GRACE).
• Evaluated model performance within subgroups defined by the unreliability score.

Main Results:

• Patients with high unreliability scores represent a subgroup with decreased model accuracy.
• This subgroup also exhibits decreased discriminatory ability of the predictive model.
• The unreliability score effectively highlights instances where model predictions may be untrustworthy.

Conclusions:

• The unreliability score provides a valuable tool for assessing the trustworthiness of individual predictions from clinical risk models.
• This method enhances clinical decision-making by identifying situations where model outputs should be interpreted with caution.
• The approach is suitable for healthcare settings, particularly those with significant class imbalance.