Gradient amplification for gradient matching based dataset distillation.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Dataset distillation (DD) aims to create compact datasets that retain the performance of models trained on larger datasets.
• Existing gradient matching (GM) frameworks for DD have limitations, primarily by not accounting for model uncertainty on incorrect labels.

Purpose of the Study:

• To develop a novel dataset distillation framework that generates more informative gradient information.
• To improve the efficiency and effectiveness of dataset distillation compared to prior methods.

Main Methods:

• Leveraging label cycle shifting with pre-trained networks to generate diverse gradients during cross-entropy loss backpropagation.
• Incorporating an early exit mechanism for faster convergence and employing an ensemble approach with exponential moving average.
• Introducing distribution matching to the total matching function to enhance gradient matching.

Main Results:

• The proposed method outperforms previous dataset distillation techniques, achieving comparable model accuracy with fewer training iterations.
• Demonstrated that gradient matching and distribution matching synergistically enhance performance.
• Experimental validation on CIFAR10, CIFAR100, TinyImageNet, and ImageNet subsets confirms the method's effectiveness.

Conclusions:

• The novel framework effectively enhances dataset distillation by utilizing label cycle shifting and improved gradient matching.
• The approach offers a more efficient and powerful method for creating distilled datasets.
• Future work could explore further optimizations and applications of this enhanced DD technique.