Expectation propagation on the diluted Bayesian classifier

Area of Science:

• Machine Learning
• Statistical Mechanics
• Data Science

Background:

• High-dimensional datasets present challenges for efficient feature selection.
• Sparse feature selection is crucial in many data-driven scientific and engineering fields.
• Existing methods like message passing and expectation maximization have limitations.

Purpose of the Study:

• To introduce a novel statistical mechanics-inspired strategy for sparse feature selection.
• To leverage expectation propagation (EP) for training a continuous-weights perceptron.
• To address binary classification problems with potentially mislabeled data.

Main Methods:

• Utilized expectation propagation (EP) for training a perceptron.
• Employed a statistical mechanics-inspired approach for feature selection.
• Tested the method in a Bayes optimal setting and compared it with existing algorithms.

Main Results:

• EP demonstrated robustness and competitiveness in variable selection, estimation accuracy, and computational complexity.
• The algorithm successfully trained from correlated patterns where other methods failed.
• EP accurately learned unknown prior parameters like dilution level and mislabeled fraction online.

Conclusions:

• Expectation propagation offers a powerful and versatile tool for sparse feature selection.
• The method provides significant advantages, particularly with complex and noisy datasets.
• EP's ability to learn parameters online enhances its practical applicability.