Cross-validation: what does it estimate and how well does it do it?

Area of Science:

• Statistics
• Machine Learning
• Computational Statistics

Background:

• Cross-validation is a standard method for estimating prediction error in machine learning.
• Its precise behavior and interpretation, particularly for linear models, are not fully understood.
• Existing methods may provide misleading estimates of prediction error and confidence intervals.

Purpose of the Study:

• To clarify what prediction error cross-validation truly estimates.
• To investigate the accuracy of confidence intervals derived from cross-validation.
• To propose improved methods for reliable prediction error estimation.

Main Methods:

• Theoretical analysis of cross-validation for linear models.
• Empirical evaluation of various prediction error estimation techniques, including data splitting and bootstrapping.
• Development and testing of a nested cross-validation scheme.

Main Results:

• Cross-validation estimates the average prediction error across different training sets, not for the specific model trained on the current data.
• Standard confidence intervals derived from cross-validation often exhibit inadequate coverage.
• Nested cross-validation provides more accurate variance estimates and reliable confidence intervals.

Conclusions:

• The interpretation of cross-validation estimates needs careful consideration.
• Nested cross-validation is a more robust approach for assessing prediction error and constructing confidence intervals.
• Re-fitting models on combined data after splitting invalidates confidence intervals.