Active Dataset Distillation via Dual-Space Informative Matching.

Area of Science:

• Machine Learning
• Computer Vision
• Artificial Intelligence

Background:

• Dataset distillation compresses large datasets into smaller synthetic ones for efficient neural network training.
• Current methods often use fixed data pools or treat all data points equally, neglecting evolving training needs.
• This can limit the effectiveness of synthetic datasets in capturing essential properties of original data.

Purpose of the Study:

• To introduce Active Dataset Distillation via Dual-Space Informative Matching (ACDD), an active learning-based approach.
• To dynamically select informative real data subsets for aligning with synthetic dataset requirements during optimization.
• To enhance training efficiency and generalization by adaptively refining the distillation pool.

Main Methods:

• ACDD employs two interconnected loops: the dual-space active loop (DAL) and the distillation loop.
• DAL dynamically selects informative samples balancing diversity and uncertainty to update the distillation pool.
• This ensures the synthetic dataset meets the evolving informational needs of the distillation process.

Main Results:

• ACDD achieves superior performance compared to state-of-the-art (SOTA) methods across multiple benchmarks (SVHN, CIFAR-10, CIFAR-100, TinyImageNet, ImageNet subset).
• The method effectively captures key characteristics of the original data within the synthetic dataset.
• ACDD reduces the required real dataset size to 20%-40% of the original, demonstrating significant efficiency gains.

Conclusions:

• ACDD offers an effective and efficient solution for dataset distillation.
• The active learning strategy dynamically adapts to training needs, improving synthetic dataset quality.
• This approach advances neural network training efficiency and generalization capabilities.