Mixture of Experts with Entropic Regularization for Data Classification.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Mixture-of-experts (MoE) models are probabilistic classifiers using expert networks and a gating network.
• Traditional MoE models can struggle with complex datasets where multiple experts are needed per data point.
• The 'winner-takes-all' nature of standard gating networks can limit model flexibility.

Purpose of the Study:

• To propose a novel variant of the mixture-of-experts model.
• To address the limitation of traditional MoE models in handling complex data requiring multi-expert influence.
• To enhance classification accuracy in real-world datasets.

Main Methods:

• Introduced a modified mixture-of-experts architecture.
• Incorporated Shannon entropy penalty into the gating network's cost function.
• Avoided the 'winner-takes-all' behavior in the gating network's output.

Main Results:

• The proposed model demonstrated improved performance on several real datasets.
• Achieved an average accuracy improvement of 3-6% on specific datasets.
• The Shannon entropy penalty effectively mitigated the 'winner-takes-all' issue.

Conclusions:

• The modified mixture-of-experts model offers a significant advantage over traditional approaches.
• The entropy-penalized gating network enhances the model's ability to capture complex data patterns.
• Future work will focus on integrating feature selection into the proposed model.