Indefinite Kernel Logistic Regression With Concave-Inexact-Convex Procedure.

Area of Science:

• Machine Learning
• Kernel Methods
• Optimization

Background:

• Kernel methods often require positive definite kernels, limiting their applicability.
• Indefinite kernels are prevalent in various real-world datasets.
• Existing kernel logistic regression (KLR) models struggle with nonpositive definite kernels.

Purpose of the Study:

• To develop an indefinite kernel learning framework for kernel logistic regression (KLR).
• To enable the effective use of nonpositive definite kernels in KLR.
• To address the limitations of standard KLR with indefinite kernels.

Main Methods:

• The proposed indefinite KLR (IKLR) model is analyzed in reproducing kernel Kreĭn spaces.
• A positive decomposition technique transforms the nonpositive definite kernel into a difference of two convex functions.
• A concave-convex procedure (CCCP) is adapted, along with its inexact (CCICP) and stochastic variants, to solve the resulting nonconvex optimization problem.

Main Results:

• The IKLR model effectively handles nonpositive definite kernels.
• The proposed concave-inexact-convex procedure (CCICP) and its stochastic variant accelerate the solving process.
• Convergence analyses are provided for both CCICP and its stochastic variant.
• Experimental results demonstrate the favorable performance of IKLR compared to standard KLR and other indefinite learning algorithms.

Conclusions:

• The developed indefinite kernel learning framework (IKLR) successfully incorporates nonpositive definite kernels into KLR.
• The proposed optimization algorithms (CCICP and stochastic CCICP) offer efficient solutions for training IKLR models.
• The IKLR model shows superior performance on benchmark datasets, highlighting its effectiveness in both deterministic and stochastic settings.