Practical selection of SVM parameters and noise estimation for SVM regression

Area of Science:

• Machine Learning
• Computational Statistics

Background:

• Support Vector Machines (SVM) are powerful tools for regression tasks.
• Hyperparameter selection, particularly for the epsilon-insensitive zone and regularization parameter C, is crucial for SVM performance.
• Current methods often rely on computationally expensive re-sampling techniques.

Purpose of the Study:

• To propose a novel analytical method for selecting SVM regression hyperparameters directly from training data.
• To demonstrate the effectiveness of this analytical approach for generalization performance.
• To compare the performance of SVM regression with epsilon-insensitive loss against other loss functions.

Main Methods:

• Developed an analytical prescription for determining the epsilon-insensitive zone value based on training sample size.
• Empirically evaluated the proposed parameter selection method on various low- and high-dimensional regression problems.
• Compared generalization performance of SVM regression with the proposed epsilon selection against least-modulus and squared loss functions.

Main Results:

• The proposed analytical hyperparameter selection method yields good generalization performance.
• SVM regression with Vapnik's epsilon-insensitive loss demonstrates superior generalization, especially with sparse samples and additive noise.
• The analytical approach avoids the need for common re-sampling methods.

Conclusions:

• Analytical hyperparameter selection for SVM regression is a practical and effective alternative to re-sampling.
• Epsilon-insensitive loss in SVM regression is vital for achieving robust performance on finite, noisy datasets.
• The proposed method offers an efficient way to optimize SVM regression models.