Efficient probabilistic classification vector machine with incremental basis function selection.

Area of Science:

• Machine Learning
• Statistical Learning Theory

Background:

• Probabilistic Classification Vector Machine (PCVM) addresses stability issues in Relevance Vector Machines for classification.
• PCVM, based on Expectation Maximization, faces challenges with initialization sensitivity, local minima convergence, and point estimates in Bayesian inference.
• Existing PCVM is inefficient for large datasets.

Purpose of the Study:

• To propose an efficient PCVM (EPCVM) that enhances training speed and stability.
• To implement EPCVM using approximation techniques for full Bayesian solutions.
• To evaluate the generalization performance and computational effectiveness of EPCVM.

Main Methods:

• EPCVM employs sequential addition/deletion of basis functions guided by marginal likelihood maximization for efficient training.
• Laplace approximation and Expectation Propagation (EP) are utilized to achieve full Bayesian inference due to the truncated prior in EPCVM.
• Hybrid Monte Carlo methods are used to validate the Laplace approximation and EP techniques.

Main Results:

• EPCVM demonstrates significant improvements in computational effectiveness for large datasets.
• The proposed method achieves strong generalization performance.
• Theoretical analysis using Rademacher complexity establishes a link between sparsity and the generalization bound of EPCVM.

Conclusions:

• EPCVM effectively addresses the limitations of traditional PCVM, offering a more stable and efficient sparse learning approach.
• The use of Laplace approximation and EP enables full Bayesian inference in EPCVM.
• EPCVM provides a robust and scalable solution for classification problems, particularly with large datasets.