Sensitivity analysis of kappa-fold cross validation in prediction error estimation

Area of Science:

• Machine Learning
• Statistical Learning Theory

Background:

• Classifier performance is typically measured by prediction error.
• Estimating prediction error is crucial in real-world machine learning problems.
• The kappa-fold cross-validation (kappa-cv) is a common error estimator.

Purpose of the Study:

• To analyze the statistical properties (bias and variance) of the kappa-cv estimator.
• To introduce a novel theoretical decomposition of kappa-cv variance.
• To compare kappa-cv bias and variance across different kappa values.

Main Methods:

• Theoretical decomposition of kappa-cv variance into sensitivity to training set and fold changes.
• Experimental evaluation using artificial domains for precise quantity computation.
• Testing with naive Bayes and nearest neighbor classifiers, varying kappa, sample size, and data distributions.

Main Results:

• The study provides a detailed breakdown of the sources contributing to kappa-cv estimator variance.
• Empirical results compare the bias-variance trade-offs for different kappa values.
• Performance analysis across various classifiers and data conditions.

Conclusions:

• The research offers a deeper understanding of kappa-cv's statistical properties.
• Practical recommendations are provided for optimal application of kappa-fold cross-validation.
• The findings aid in selecting appropriate error estimation techniques in machine learning.