Alternative performance measures for prediction models.

Area of Science:

• Biostatistics
• Machine Learning
• Medical Informatics

Background:

• The area under the receiver operating characteristic curve (AUC) is a common performance measure for prediction models but can be insensitive to improvements from new markers, especially in the "gray zone."
• Recently proposed relative-performance measures may yield contradictory conclusions.
• Evaluating prediction model performance requires sensitive and interpretable metrics.

Purpose of the Study:

• To compare the sensitivity of various performance measures to changes in prediction model performance when a new marker is added.
• To identify alternative measures that are more sensitive to improvements in the "gray zone" of prediction accuracy.

Main Methods:

• Computer simulations were used to assess the performance of different measures.
• The study evaluated the area under the receiver operating characteristic curve (AUC), Gini index, Pietra index, and scaled Brier score.
• Simulations focused on scenarios where a new marker's discrimination power was concentrated in the "gray zone" of a baseline model.

Main Results:

• The area under the receiver operating characteristic curve (AUC) and the Gini index showed minimal performance improvements when the added marker's power was in the "gray zone."
• The Pietra index and the scaled Brier score demonstrated more significant performance improvements in the same "gray zone" scenario.
• These findings highlight differences in sensitivity among various prediction model performance metrics.

Conclusions:

• The Pietra index and the scaled Brier score are recommended for prediction model performance measurement.
• These measures offer superior sensitivity to markers that improve discrimination in the "gray zone" compared to AUC and Gini.
• Ease of interpretation and clinical relevance further support the use of the Pietra and scaled Brier indices.