Misuse of DeLong test to compare AUCs for nested models

Area of Science:

• Biostatistics
• Epidemiology
• Medical Statistics

Background:

• The area under the receiver operating characteristics curve (AUC of ROC) is crucial for evaluating risk prediction models.
• The Mann-Whitney statistic estimates AUC, and the DeLong test assesses changes in AUC.
• A common issue arises when a new predictor improves model discrimination but the DeLong test fails to detect significant AUC improvement in nested models.

Purpose of the Study:

• To investigate the reasons behind the DeLong test's failure to detect significant AUC improvements in nested models.
• To provide a theoretical explanation and simulation-based evidence for this statistical phenomenon.

Main Methods:

• Theoretical analysis using generalized U-statistics.
• Numerical simulations across various scenarios with nested models.
• Comparison of DeLong test performance with predictor association and non-nested models.

Main Results:

• The DeLong test's assumption of normality is violated when the added predictor is not significantly associated with the outcome in nested models.
• This misuse leads to a loss of statistical power, resulting in a conservative test for small to moderate effect sizes.
• The issue is specific to nested models and does not affect non-nested models or predictors strongly associated with the outcome.

Conclusions:

• The DeLong test is inappropriately applied for assessing AUC changes in nested models when the added predictor lacks significant association.
• For nested models, it is recommended to first perform a test of association for new predictors.
• If the association is significant, estimate the change in AUC using an appropriate confidence interval, potentially based on the DeLong approach.