Testing the Relative Performance of Data Adaptive Prediction Algorithms: A Generalized Test of Conditional Risk...

Area of Science:

• Statistics
• Biostatistics
• Genetics

Background:

• Comparing model fit is crucial for scientific questions.
• Traditional methods like likelihood ratio tests and C-statistics are unsuitable for data-adaptive procedures.
• Cross-validation is often used but poses inferential challenges.

Purpose of the Study:

• To propose a general statistical approach for assessing the relative fit of competing models, particularly for data-adaptive methods.
• To develop a Wald-type test statistic and confidence intervals for cross-validated test sets.
• To provide a reliable method for evaluating model improvement in prediction risk.

Main Methods:

• Development of a general approach focusing on the conditional risk difference.
• Derivation of a Wald-type test statistic and confidence intervals using independent validation within cross-validation.
• Inclusion of a test for multiple comparisons to maintain Type I error control.

Main Results:

• The proposed test maintains proper Type I Error under the null hypothesis.
• The method is applicable as a general test of relative fit for any semi-parametric model alternative.
• Demonstrated application in a candidate gene study for pathway analysis.

Conclusions:

• The new statistical test provides a robust method for comparing model fits, overcoming limitations of traditional approaches.
• It enables reliable inference for data-adaptive models and semi-parametric alternatives.
• The approach is validated through application in genetic association studies.