Specific Expert Learning: Enriching Ensemble Diversity via Knowledge Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Ensemble methods are popular for visual tasks but limited by model diversity.
• Lack of diversity in ensembles restricts overall performance.
• Existing methods struggle to effectively increase ensemble diversity.

Purpose of the Study:

• To introduce a novel knowledge distillation approach, Learning from Experts (LFEs), to enhance ensemble diversity.
• To develop Specific Expert Learning (SEL) to improve performance on weaker classes and overall accuracy.
• To demonstrate the effectiveness of SEL in creating diverse ensembles and improving classifier accuracy.

Main Methods:

• Proposed Specific Expert Learning (SEL), a novel knowledge distillation (KD) method.
• SEL enables models to acquire diverse knowledge from networks with varied expertise.
• Applied SEL to single classifiers and ensemble distillation (ED).

Main Results:

• SEL increased ResNet-32 accuracy by 0.91% on CIFAR-10.
• Ensembles trained with SEL achieved a 1.13% accuracy increase.
• SEL outperformed state-of-the-art methods like DML in accuracy improvements.

Conclusions:

• Specific Expert Learning (SEL) effectively enhances the diversity of ensemble models.
• SEL improves the accuracy of individual classifiers and ensemble performance.
• The proposed approach offers a significant advancement in ensemble learning for visual tasks.