Least square regularized regression in sum space.

Area of Science:

• Machine Learning
• Functional Analysis
• Numerical Analysis

Background:

• Reproducing Kernel Hilbert Spaces (RKHSs) are powerful tools for function approximation.
• Nonflat function approximation presents challenges due to varying frequency components.
• Existing methods may not optimally handle complex function landscapes.

Purpose of the Study:

• To propose a least square regularized regression algorithm in the sum space of RKHSs for nonflat function approximation.
• To analyze the convergence and learning rates of the proposed algorithm.
• To demonstrate the superiority of the sum space approach over single RKHS methods.

Main Methods:

• Developed a least square regularized regression algorithm operating in the sum space of RKHSs.
• Approximated target functions by decomposing them into low- and high-frequency components using large and small scale kernels.
• Analyzed algorithm complexity using covering numbers and bounded them by the product of covering numbers of basic RKHSs.
• Solved the algorithm by addressing a system of linear equations.

Main Results:

• The algorithm effectively approximates both low- and high-frequency components of target functions.
• A polynomial learning rate was achieved for sum spaces of RKHSs with Gaussian kernels.
• The proposed method demonstrated a better learning rate than any single RKHS.
• The utility was validated on simulated and real-life datasets.

Conclusions:

• The sum space of RKHSs provides a more effective framework for nonflat function approximation.
• The proposed algorithm offers a tunable tradeoff between sample and regularization errors.
• This approach yields improved learning rates and demonstrates practical applicability.