Sel4FT: Annotation Selection for Pretraining-Finetuning With Distribution Shift.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• The pretraining-finetuning paradigm is standard in computer vision.
• Efficiently utilizing limited annotation budgets during finetuning is a critical challenge.
• Existing methods lack effective strategies for sample selection under budget constraints.

Purpose of the Study:

• Introduce active finetuning as a novel task for optimal sample selection.
• Propose Sel4FT, a unified framework for selecting informative and diverse data subsets.
• Develop Sel4FT++ to handle distribution shifts caused by data augmentation.

Main Methods:

• Sel4FT optimizes a parametric model in continuous feature space to select subsets.
• The selection process preserves the full data pool's distribution and maintains diversity.
• Sel4FT++ incorporates augmentation-aware mechanisms to address distribution shifts.
• Theoretical analysis proves minimization of Earth Mover's Distance between subset and full pool.

Main Results:

• Sel4FT and Sel4FT++ achieve state-of-the-art performance across various tasks.
• Demonstrated effectiveness in image classification, long-tailed recognition, and semantic segmentation.
• Achieved over 100x speedup compared to existing annotation selection methods.
• Eliminates iterative retraining and annotation during the selection process.

Conclusions:

• Active finetuning with Sel4FT offers an efficient solution for real-world deployment.
• The framework significantly enhances model performance while optimizing annotation budgets.
• Sel4FT provides a robust and scalable approach to data selection in computer vision.