Net reclassification index at event rate: properties and relationships.

Area of Science:

• Biostatistics
• Medical Informatics
• Epidemiology

Background:

• Net reclassification improvement (NRI) is a common metric for assessing prediction model performance.
• Existing NRI methods were primarily developed for models with fixed classification thresholds.
• The application of NRI has expanded to scenarios lacking established thresholds.

Purpose of the Study:

• To explore the properties of the net reclassification improvement (NRI) at the event rate.
• To demonstrate the relationship between NRI at event rate and other model performance measures.
• To provide a framework for evaluating incremental model improvement.

Main Methods:

• Expressing NRI at event rate as a difference in performance metrics between new and old models.
• Relating NRI at event rate to global and decision-analytic measures of model performance.
• Demonstrating its connection to maximizing relative utility and Kolmogorov-Smirnov distance.

Main Results:

• The NRI at event rate is shown to be a measure that maximizes relative utility across all classification thresholds.
• It quantifies the reduction in expected regret, serving as a criterion based on the value of information.
• The study presents plots of standardized net benefit to visualize model performance increments.

Conclusions:

• The NRI at event rate offers a robust method for evaluating incremental improvements in prediction models, especially in threshold-independent settings.
• Visualizing model performance through standardized net benefit plots enhances understanding of model utility.
• The findings are validated through theoretical examples and a clinical application in atrial fibrillation risk prediction.