Related Experiment Videos
Trading variance reduction with unbiasedness: the regularized subspace information criterion for robust model
Masashi Sugiyama1, Motoaki Kawanabe, Klaus-Robert Müller
1Fraunhofer FIRST, IDA, 12489 Berlin, Germany. sugi@cs.titech.ac.jp
Neural Computation
|April 9, 2004
Summary
This study stabilizes unbiased generalization error estimates using regularization, improving model selection. By minimizing squared error, the proposed method enhances the precision of the subspace information criterion (SIC), especially in high-noise scenarios.
Area of Science:
- Machine Learning
- Statistical Learning Theory
Background:
- Biased estimators can outperform unbiased ones in specific scenarios.
- The subspace information criterion (SIC) is an unbiased estimator for expected generalization error in kernel regression.
- Regularization of SIC has shown stabilization effects, but optimal degree determination was unclear.
Purpose of the Study:
- To stabilize unbiased generalization error estimates via regularization.
- To develop a robust model selection criterion for learning.
- To determine the optimal regularization degree for SIC.
Main Methods:
- Deriving an unbiased estimator for the expected squared error between SIC and expected generalization error.
- Proposing a method to determine SIC regularization by minimizing this derived estimator.
- Utilizing computer simulations with artificial and real datasets.
Main Results:
- The proposed regularization method effectively improves SIC precision.
- Improvements are particularly notable in high-noise level cases.
- The method demonstrates competitive performance against cross-validation and empirical Bayesian methods.
Conclusions:
- The novel approach provides a principled way to regularize SIC for enhanced model selection.
- This method offers improved robustness and precision in generalization error estimation.
- The findings contribute to more reliable machine learning model selection, especially under noisy conditions.