Fast Cross-Validation for Kernel-Based Algorithms.

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Science

Background:

• Cross-validation (CV) is essential for model selection.
• Empirical cross-validation error (CVE) computation is computationally intensive due to repeated model training.

Purpose of the Study:

• To develop a novel approximation theory for CVE.
• To present an approximate CV approach for kernel-based algorithms using the Bouligand influence function (BIF).

Main Methods:

• Representing BIF and higher-order BIFs using Taylor expansions.
• Deriving an upper bound for the discrepancy between original and approximate CV.
• Developing a novel method to compute BIF for general distributions.

Main Results:

• The proposed approximate CV trains on the full dataset only once.
• The method is applicable to a wide range of kernel-based algorithms.
• Experimental results confirm the soundness and effectiveness of the approximate CV.

Conclusions:

• The novel approximate CV method offers a computationally efficient alternative to traditional CV.
• This approach maintains accuracy while significantly reducing training time.
• The BIF-based approximation is a promising direction for efficient model selection in kernel methods.