ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Model updates in classification systems can introduce errors known as negative flips.
• Current methods to reduce the negative flip rate (NFR) either decrease overall accuracy or significantly increase inference costs using ensembles.

Purpose of the Study:

• To develop a novel method for reducing NFR while maintaining high classification accuracy.
• To achieve these improvements at the inference cost of a single model.

Main Methods:

• Analysis of ensemble behavior in reducing NFR, identifying that they target flips with large logit deviations.
• Introduction of Ensemble Logit Difference Inhibition (ELODI), a method that distills a homogeneous ensemble into a single student model.
• Development of a generalized distillation objective, Logit Difference Inhibition (LDI), which selectively penalizes logit differences for high-logit classes.

Main Results:

• ELODI successfully trains a single model that matches ensemble performance in NFR reduction.
• The method demonstrates superior accuracy retention compared to existing approaches.
• Experiments on image classification benchmarks confirm significant NFR reduction and accuracy preservation.

Conclusions:

• ELODI offers an efficient solution for mitigating negative flips in classification systems.
• The approach balances error reduction and accuracy, overcoming limitations of prior methods.
• This technique enables cost-effective model updates with improved performance.