Kernel learning at the first level of inference.

Area of Science:

• Machine Learning
• Computational Statistics

Background:

• Kernel learning methods involve multi-level inference for parameter tuning.
• Model selection in kernel machines risks overfitting, especially with many parameters like in Automatic Relevance Determination (ARD).
• Conventional methods often rely on cross-validation or theoretical bounds for parameter tuning.

Purpose of the Study:

• To investigate learning kernel parameters at the first inference level for Least-Squares Support Vector Machine (LS-SVM) classifiers.
• To jointly optimize kernel parameters and expansion coefficients, minimizing a regularized training criterion.
• To alleviate the problem of overfitting the model selection criterion by reducing the number of parameters to tune.

Main Methods:

• Joint optimization of kernel parameters and kernel expansion coefficients.
• Introduction of a regularisation term acting on kernel parameters.
• Evaluation on synthetic and real-world binary classification benchmark problems.

Main Results:

• Kernel learning at the first inference level demonstrates statistically superior performance compared to conventional approaches.
• The proposed method significantly alleviates the risk of overfitting the model selection criterion.
• The approach is competitive with Multiple Kernel Learning but offers reduced computational cost.

Conclusions:

• Learning kernel parameters at the first inference level is an effective strategy for LS-SVM classifiers.
• This method improves generalization performance by mitigating model selection overfitting.
• The approach offers a computationally efficient and statistically robust alternative to existing kernel learning techniques.