Revisiting performance metrics for prediction with rare outcomes

Area of Science:

• Clinical informatics
• Biostatistics
• Machine learning in medicine

Background:

• Machine learning (ML) algorithms are increasingly adopted in clinical research, often presented as superior to traditional methods like logistic regression.
• Commonly used performance metrics, such as area under the receiver operating characteristic curve (AUC), may provide a misleading evaluation of binary prediction models, especially for rare outcomes.

Purpose of the Study:

• To illustrate the potential pitfalls of relying on single performance metrics, particularly AUC, in evaluating binary prediction models within a realistic clinical context.
• To highlight the challenges and appropriate evaluation strategies for predicting rare post-surgery medical complications.

Main Methods:

• Prediction of post-surgery mortality in patients undergoing aortic valve replacement using a clinical registry.
• Incorporation of multiple evaluation metrics and ML algorithms suitable for rare outcomes, including ensemble methods and a novel lasso extension.
• Cross-validation was employed to assess model performance.

Main Results:

• All tested algorithms demonstrated high overall accuracy and moderate cross-validated AUC values.
• Despite a low false positive rate (<1%) and high positive predictive value (100%), the true positive rate for mortality prediction was critically low (7%).
• Calibration plots revealed poor model calibration, and simulations confirmed that high AUC can accompany low true positive rates.

Conclusions:

• Clinical studies should avoid relying exclusively on AUC or accuracy for evaluating binary prediction models, especially when dealing with rare events.
• A comprehensive set of evaluation metrics is crucial for a realistic assessment of model performance and clinical utility in medical research.