Forget Me Not: Fighting Local Overfitting With Knowledge Fusion and Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) exhibit less overfitting than theoretical predictions suggest.
• Conventional overfitting, a global performance decline with increased capacity, is rarely observed in practice.
• The study investigates overfitting occurring in specific data sub-regions, termed local overfitting.

Purpose of the Study:

• Introduce a novel score to measure the forgetting rate of DNNs on validation data.
• Define and quantify local overfitting as performance degradation in specific input space regions.
• Explore the link between local overfitting and the double descent phenomenon.

Main Methods:

• Developed a novel score to quantify the forgetting rate on validation data.
• Proposed a two-stage approach: checkpoint aggregation into an ensemble, followed by knowledge distillation.
• Leveraged the training history of a single model to recover forgotten knowledge.

Main Results:

• Demonstrated that local overfitting can occur independently of conventional overfitting.
• Showed a strong correlation between local overfitting and the double descent phenomenon.
• The proposed Knowledge Fusion followed by Knowledge Distillation method enhanced performance without increasing inference cost, outperforming baselines, especially with label noise.

Conclusions:

• Local overfitting is a distinct phenomenon from global overfitting, linked to model capacity and training dynamics.
• A novel two-stage method effectively recovers and retains forgotten knowledge from a model's training history.
• This approach offers improved performance and reduced complexity, presenting a win-win scenario for deep learning model optimization.