Curriculum Dataset Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Dataset distillation methods face challenges with large-scale datasets due to high computational and memory demands.
• Existing scalable disentanglement methods show promise but have performance bottlenecks.
• There is a need for optimized, scalable dataset distillation techniques.

Purpose of the Study:

• To present a curriculum-based dataset distillation framework that balances performance and scalability for large datasets.
• To address the homogeneity and simplicity issues in previously generated distilled images.
• To enhance the generalization and robustness of distilled datasets.

Main Methods:

• Developed a curriculum-based framework for strategic distillation of synthetic images, progressing from simple to complex.
• Incorporated curriculum evaluation to improve image diversity and reduce computational cost.
• Utilized adversarial optimization for synthetic images to boost representativeness and prevent overfitting.

Main Results:

• Achieved new benchmarks in large-scale dataset distillation.
• Demonstrated substantial performance improvements: 11.1% on Tiny-ImageNet, 9.0% on ImageNet-1K, and 7.3% on ImageNet-21K.
• Enhanced generalization across various neural network architectures and improved robustness to noise.

Conclusions:

• The proposed curriculum-based dataset distillation framework effectively harmonizes performance and scalability.
• The method overcomes limitations of previous approaches, producing more representative and generalizable distilled datasets.
• This framework offers a promising solution for efficient and effective large-scale dataset distillation.