Measures for evaluation of prognostic improvement under multivariate normality for nested and nonnested models

Area of Science:

• Biostatistics
• Medical Informatics
• Epidemiology

Background:

• Comparing risk prediction models requires metrics to quantify prognostic improvement.
• Existing metrics include changes in AUC, IDI, NRI, SNB, Brier score, and scaled Brier score.

Purpose of the Study:

• To explore the behavior and interrelationships of six prognostic improvement metrics.
• To demonstrate the utility of theoretical estimators for these metrics, particularly under normality.
• To assess metric performance with non-normal data using a real-world example.

Main Methods:

• Analysis under multivariate normality using linear discriminant analysis.
• Demonstration of metrics as functions of squared Mahalanobis distance.
• Simulation studies comparing theoretical and empirical estimation methods.
• Application to Framingham Heart Study cardiovascular disease risk models.

Main Results:

• All six metrics are functions of squared Mahalanobis distance, a measure of discrimination.
• Theoretical estimators under normality perform comparably or superiorly to empirical methods.
• Theoretical estimators for Net Reclassification Index and Standardized Net Benefit show reduced variability.

Conclusions:

• Squared Mahalanobis distance provides a unifying framework for understanding these prognostic metrics.
• Theoretical estimation offers a more stable alternative for Net Reclassification Index and Standardized Net Benefit.
• Findings enhance interpretability and clinical utility of risk prediction model comparisons.