Bridging the Gap Between Active Learning and Pseudolabel Learning for Vision-Language Models

Recent studies have explored enhancing label efficiency in vision-language models (VLMs) for downstream tasks. Existing methods of pseudolabel learning (PL) would suffer from heavily incorrect hard pseudolabels. Nevertheless, active learning (AL) focuses on uncertain samples that can cause forgetting of VLMs' existing knowledge. Can these two paradigms work together to enhance the performance of VLMs? To alleviate the issues mentioned above, we bridge the gap between the two via a resampling mechanism that balances the data distribution without increasing labeling costs. By offering a comprehensive and well-distributed labeled set, the downstream task models based on VLMs receive balanced gradient updates during training, which helps to reduce miscalibration and overfitting in models. Comprehensive experimental results demonstrate the effectiveness and general applicability of our method compared to state-of-the-art label-efficient approaches with VLMs across several benchmark datasets.