Nonlinear kernel-based statistical pattern analysis.

Area of Science:

• Machine Learning
• Pattern Analysis
• Statistical Inference

Background:

• Second-order statistics of multivariate populations are crucial for pattern analysis.
• Kernel functions implicitly map data to high-dimensional feature spaces, enabling nonlinear analysis.
• Standard algorithms like PCA and Fisher's discriminant are typically linear.

Purpose of the Study:

• To derive general expressions for nonlinear counterparts of standard pattern analysis algorithms using kernel functions.
• To introduce kernel versions of Mahalanobis distance and minimum squared error (MSE) linear discriminant functions.
• To explore the generalization properties of kernel machines in comparison to Support Vector Machines (SVMs).

Main Methods:

• Inferred eigenstructure from pairwise inner products in implicit, high-dimensional feature spaces defined by kernels.
• Developed general expressions for nonlinear extensions of principal component analysis, data compression, denoising, and Fisher's discriminant.
• Introduced kernel versions of Mahalanobis distance and MSE linear discriminant functions, including potential functions and RBF networks.

Main Results:

• Established a connection between kernel methods and nonparametric density estimation.
• Kernel Mahalanobis distance yields nonparametric models with novel properties.
• Kernel MSE linear discriminant functions are simple and encompass generalized linear models.
• Demonstrated that simple kernel machines based on pseudoinversion are competitive with SVMs, especially in cases of significant class overlap.

Conclusions:

• Kernel methods provide a powerful framework for extending linear pattern analysis techniques into nonlinear domains.
• The proposed kernel machines offer efficient and competitive alternatives to existing methods like SVMs.
• These findings offer insights into the interplay of feature spaces and inductive bias in machine learning generalization.