Soft Label Pruning and Quantization for Large-Scale Dataset Distillation.

Area of Science:

• Computer Vision
• Machine Learning
• Data Compression

Background:

• Large-scale dataset distillation necessitates substantial storage for auxiliary soft labels, often hundreds of times larger than condensed images.
• Existing methods are hindered by insufficient image diversity and supervision diversity, leading to performance degradation at high compression rates.

Purpose of the Study:

• To address the storage and performance issues in large-scale dataset distillation.
• To propose a novel method, Label Pruning and Quantization for Large-scale Distillation (LPQLD), for efficient dataset compression.

Main Methods:

• Enhancing image diversity through class-wise batching and Batch Normalization (BN) supervision during synthetic data generation.
• Improving supervision diversity via Label Pruning with Dynamic Knowledge Reuse and Label Quantization with Calibrated Student-Teacher Alignment.

Main Results:

• Reduced soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K.
• Achieved accuracy improvements of up to 7.2% on ImageNet-1K and 2.8% on ImageNet-21K.
• Demonstrated superiority across various network architectures and compared to other distillation methods.

Conclusions:

• LPQLD effectively overcomes the limitations of large soft label storage in dataset distillation.
• The proposed method achieves significant compression ratios while enhancing model accuracy.
• LPQLD represents a superior approach for efficient large-scale dataset distillation.