Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted...

Area of Science:

• Biostatistics
• Epidemiology
• Medical Informatics

Background:

• Prediction models are crucial in healthcare for assessing disease risk.
• The area under the receiver operating characteristic curve (AUC) is a common metric for model performance.
• Minimal AUC improvements from adding risk factors are often considered clinically insignificant.

Purpose of the Study:

• To investigate if risk factors that minimally improve AUC can still enhance predictive ability.
• To assess predictive improvement using integrated discrimination improvement (IDI).

Main Methods:

• Simulated 100,000 individual datasets with varying baseline AUCs (0.50-0.95).
• Introduced single risk factors modeled by odds ratios (OR 2, 4, 8) or AUC increments (ΔAUC 0.01, 0.02, 0.03).
• Evaluated changes in AUC and IDI.

Main Results:

• Both ΔAUC and IDI decreased as baseline AUC increased for a given OR.
• Small AUC increments (ΔAUC 0.01) resulted in small IDI, except when baseline AUC exceeded 0.90.
• Minimal AUC improvement generally led to minimal changes in predicted risks.

Conclusions:

• Risk models updated with strong risk factors may benefit specific subgroups but not the general population.
• The AUC metric might be more sensitive to improvements than commonly perceived.
• Integrated discrimination improvement (IDI) offers a complementary view on model enhancement beyond AUC.