Generalization Properties of Robust Learning With Random Features.

Area of Science:

• Machine Learning
• Statistical Learning Theory
• Robust Statistics

Background:

• Random features (RFs) offer computational efficiency for kernel methods on large datasets.
• Standard RF algorithms are sensitive to outliers and heavy-tailed noise, degrading performance.
• Robustness is crucial for real-world machine learning applications with imperfect data.

Purpose of the Study:

• To develop a robust and adaptive regularized least squares method with random features (RRLS-RF).
• To address the performance degradation of standard RFs caused by data contamination.
• To establish theoretical guarantees and demonstrate practical advantages of the proposed method.

Main Methods:

• Proposed a robust and adaptive regularized least squares method with random features (RRLS-RF).
• Incorporated response truncation with an adaptively determined truncation level.
• Established generalization properties under weak moment conditions (bounded $(1+\delta)$-th moment).

Main Results:

• RRLS-RF achieves learning rates of $\mathcal {O}(|D|^{-\frac{\delta }{2\delta +2}})$ with $\mathcal {O}(|D|^{\frac{\delta }{2\delta +2}}\log |D|)$ random features.
• Theoretical rates converge to optimal $\mathcal {O}(|D|^{-\frac{1}{2}})$ as $\delta \rightarrow \infty$, surpassing standard assumptions.
• Demonstrated optimality via minimax lower bounds and superior robustness in numerical experiments.

Conclusions:

• RRLS-RF provides a robust and theoretically sound alternative to standard RF methods for noisy datasets.
• The adaptive truncation strategy effectively balances robustness and bias.
• The method shows practical advantages in handling outliers and heavy-tailed noise, verified by experiments.