Testing for improvement in prediction model performance.

Area of Science:

• Biostatistics
• Epidemiology
• Medical Informatics

Background:

• Evaluating new predictors in risk models is crucial for improving prediction performance.
• Existing methodologies for assessing prediction improvement can be complex and lead to redundant testing.

Purpose of the Study:

• To theoretically prove the equivalence between testing for prediction improvement and testing for risk factor significance.
• To investigate the properties of statistical tests for prediction improvement, particularly the area under the ROC curve (AUC).
• To identify reasons for the perceived insensitivity of AUC to prediction performance improvements.

Main Methods:

• Theoretical derivation of null hypothesis equivalence.
• Simulation studies to evaluate statistical test properties.
• Analysis of standard testing procedures for regression models.

Main Results:

• Null hypotheses for prediction improvement are equivalent to testing if a new predictor is a risk factor.
• Standard inference procedures without adjustments for regression coefficient variability are extremely conservative.
• The insensitivity of AUC may stem from invalid inference methods, not the measure itself.

Conclusions:

• Hypothesis testing for prediction improvement is redundant when the predictor's risk factor status is established.
• Recommend focusing on estimation of prediction performance measures over hypothesis testing.
• Advise using well-developed methods for evaluating risk factors to avoid redundant and problematic inference procedures.