Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates.

Area of Science:

• Machine Learning
• Statistical Modeling
• Predictive Analytics

Background:

• Area Under the ROC Curve (AUC) is a standard metric for binary classification model performance.
• Cross-validation is frequently used with AUC to assess generalization to new data.
• Estimating the variance of cross-validated AUC is essential for evaluating estimate quality.

Purpose of the Study:

• To develop a computationally efficient method for estimating the variance of cross-validated AUC.
• To address the computational intractability of bootstrapping for large datasets or complex models.
• To provide a practical alternative for variance estimation in machine learning model evaluation.

Main Methods:

• Influence curve methodology applied to cross-validated AUC.
• Development of a computationally efficient algorithm for variance estimation.
• Comparison with traditional bootstrapping methods (where feasible).

Main Results:

• The proposed influence curve approach provides a computationally efficient variance estimate for cross-validated AUC.
• This method is particularly advantageous for massive datasets and complex prediction models where bootstrapping is infeasible.
• Demonstrated the accuracy and efficiency of the influence curve method.

Conclusions:

• The influence curve based approach offers a computationally tractable solution for variance estimation of cross-validated AUC.
• This method enhances the practical evaluation of predictive model performance, especially in resource-constrained settings.
• Facilitates more reliable assessment of model generalizability through efficient variance estimation.