Improved small-sample estimation of nonlinear cross-validated prediction metrics.

Area of Science:

• Statistics
• Machine Learning
• Predictive Modeling

Background:

• Evaluating prediction algorithm performance using training data presents challenges.
• Common methods such as bootstrapping and cross-validation have known limitations and can yield suboptimal results.
• There is a need for robust and accurate performance estimation techniques in machine learning.

Purpose of the Study:

• To investigate the shortcomings of existing methods for evaluating prediction algorithms on training data.
• To propose a novel framework for performance estimation based on efficiency theory.
• To provide theoretical guarantees and demonstrate practical improvements over standard approaches.

Main Methods:

• A general theoretical study of cross-validation-based estimators.
• Development of a new estimation framework drawing from efficiency theory.
• Mathematical proof of weak convergence for the proposed estimators.
• Empirical evaluation on two specific prediction tasks.

Main Results:

• Bootstrap-based performance estimation methods were shown to frequently fail.
• Standard cross-validation estimators were found to perform poorly in certain scenarios.
• The proposed efficiency theory-based estimators demonstrated significant finite-sample improvements.
• Theoretical guarantees of weak convergence were established for the new estimators.

Conclusions:

• Existing methods for evaluating prediction algorithms on training data are often unreliable.
• The proposed alternative framework offers a more robust and accurate approach to performance estimation.
• The new estimators provide substantial finite-sample improvements, enhancing the reliability of predictive models.