Toward Mitigating Architecture Overfitting on Distilled Datasets.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Dataset distillation (DD) enhances neural network training with limited data.
• A key challenge is architecture overfitting, where distilled datasets perform poorly with different network architectures, especially larger ones.

Purpose of the Study:

• To introduce and validate methods for mitigating architecture overfitting in dataset distillation.
• To improve the generalizability of distilled datasets across various network architectures.

Main Methods:

• Employing DropPath to create implicit ensembles of subnetworks within larger models.
• Utilizing knowledge distillation (KD) to align subnetwork behavior with a well-performing teacher network.
• Characterizing these methods by their smoothing effects on the distilled data.

Main Results:

• Extensive experiments demonstrate significant mitigation of architecture overfitting across diverse tasks and dataset sizes.
• Proposed methods achieve comparable or superior performance even when test networks have greater capacity than training networks.
• Validated the effectiveness and generality of the developed approaches.

Conclusions:

• The introduced methods effectively address architecture overfitting in dataset distillation.
• These techniques enhance the robustness and applicability of DD across different neural network architectures.
• The findings pave the way for more versatile and reliable dataset distillation techniques.