Understanding increments in model performance metrics.

Area of Science:

• Biostatistics
• Epidemiology
• Medical Informatics

Background:

• The area under the receiver operating characteristic curve (AUC) is a standard measure for binary outcome prediction models.
• Recent criticisms highlight AUC's limitations in detecting improvements when adding predictors to well-discriminating models.
• This raises concerns about missing clinical performance gains from updated risk prediction rules.

Purpose of the Study:

• To investigate the claim that AUC may miss improvements in clinical performance.
• To compare AUC with discrimination slope in predicting changes in sensitivity and specificity.
• To evaluate how model discrimination impacts the required predictor strength for AUC increments.

Main Methods:

• Relating AUC to clinical performance measures (sensitivity, specificity) under multivariate normality.
• Contrasting AUC behavior with discrimination slope.
• Illustrating theoretical findings with a Framingham Heart Study dataset for atrial fibrillation prediction.

Main Results:

• AUC change adequately predicts clinical performance changes, except when excellent specificity is crucial.
• Achieving the same AUC increment requires stronger or more predictors for models with good discrimination compared to poor discrimination.
• Discrimination slope may be a superior measure for model improvement when high specificity is desired.

Conclusions:

• AUC is generally a suitable metric for assessing prediction model improvements.
• The discrimination slope offers advantages over AUC when excellent specificity is a primary goal.
• Model discrimination influences the effectiveness of adding predictors for improving AUC.