Small Sample Inference for Generalization Error in Classification Using the CUD Bound.

Area of Science:

• Machine Learning
• Statistical Learning Theory

Background:

• Estimating generalization error is critical for classifier performance, especially with limited training data.
• Existing methods using resampling and distribution assumptions often fail to provide accurate confidence sets due to the non-smooth nature of generalization error.
• The non-normal distribution of estimated generalization error complicates reliable confidence interval construction.

Purpose of the Study:

• To develop a robust method for constructing confidence sets for generalization error in machine learning classifiers.
• To address the limitations of existing techniques that struggle with the inherent non-smoothness of generalization error.
• To provide a computationally efficient algorithm for parametric additive models.

Main Methods:

• Constructing a confidence set for generalization error using a smooth upper bound on the deviation between resampled estimates and true error.
• Bootstrapping this smooth upper bound to form the confidence set.
• Developing a computationally efficient algorithm for classifiers represented as parametric additive models.

Main Results:

• The proposed method demonstrates superior performance in constructing confidence sets for generalization error compared to traditional approaches.
• The technique effectively handles the non-smoothness issue, leading to more reliable confidence intervals.
• Validated through performance on test and simulated datasets.

Conclusions:

• The novel bootstrapping approach using a smooth upper bound offers a reliable solution for confidence measures in generalization error estimation.
• This method improves the accuracy and trustworthiness of confidence sets, particularly for small training sample sizes.
• The computationally efficient algorithm enhances practical applicability in specific classifier settings.