A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels.

Area of Science:

• Machine Learning
• Kernel Methods
• Bayesian Nonparametrics

Background:

• Random Fourier Features (RFFs) are effective for kernel approximation but limited by Bochner's theorem, excluding essential kernels like dot-product and indefinite kernels.
• Existing RFF methods struggle with kernels that are not positive definite (PD) or shift-invariant, restricting their applicability in large-scale machine learning.
• Indefinite kernels, common in various applications, require specialized approximation techniques beyond standard RFFs.

Purpose of the Study:

• To develop a unified Random Fourier Feature (RFF) framework capable of approximating indefinite kernels within reproducing kernel Kreĭn spaces (RKKSs).
• To extend RFF applicability to dot-product kernels on the unit sphere by transforming them into indefinite kernels.
• To provide a flexible and scalable method for approximating a wide range of indefinite kernels.

Main Methods:

• Utilizing the Kolmogorov decomposition scheme to represent indefinite kernels as a difference of two positive definite (PD) kernels.
• Formulating the spectral distribution of underlying PD kernels using a nonparametric Bayesian Gaussian mixtures model.
• Proposing a double-infinite Gaussian mixture model with a Dirichlet process prior for flexible RFF approximation.
• Developing a non-conjugate variational inference algorithm with a sub-sampling scheme for efficient posterior inference.

Main Results:

• The proposed double-infinite Gaussian mixture model effectively approximates indefinite kernels with high flexibility in the number of components.
• The non-conjugate variational inference with sub-sampling provides an efficient approach for model training on large datasets.
• Experimental results demonstrate superior performance of the proposed method for indefinite kernel approximation compared to existing RFF techniques on large classification datasets.

Conclusions:

• The developed RFF framework offers a unified and effective solution for approximating indefinite kernels in RKKSs.
• This approach significantly broadens the applicability of RFFs to a wider class of kernels, including dot-product and indefinite kernels.
• The nonparametric Bayesian approach provides a scalable and flexible tool for large-scale machine learning tasks involving complex kernel approximations.