Sel4FT: Annotation Selection for Pretraining-Finetuning With Distribution Shift

The pretraining-finetuning paradigm has become dominant in computer vision, yet strategically exploiting limited annotation budgets during finetuning remains unexplored. We introduce active finetuning-a novel task for selecting the most informative samples to annotate within this paradigm. We propose Sel4FT, a unified annotation selection framework that optimizes a parametric model in continuous feature space to identify a subset preserving the entire pool's distribution while maintaining diversity. To address distribution shifts from data augmentation, we develop Sel4FT++ with augmentation-aware selection mechanisms. We theoretically prove our approach minimizes the Earth Mover's Distance between selected subset and full data pool. Our framework eliminates iterative retraining and annotation process during selection, providing an efficient solution for real-world deployment. Extensive experiments on image classification, long-tailed recognition, and semantic segmentation demonstrate state-of-the-art performance with over $100\times$100× speedup compared to existing methods.