Synth-CLIP: Synthetic data make CLIP generalize better in data-limited scenarios

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Prompt learning effectively transfers Vision-Language Models (VLMs) to downstream tasks.
• Fine-tuning VLMs solely on base classes hinders generalization to novel classes, especially with limited data.

Purpose of the Study:

• To propose Synth-CLIP, an innovative approach leveraging synthetic data to enhance VLM generalization for both base and novel classes.
• To improve the capability of pre-trained models like CLIP to adapt to new categories without visual samples.

Main Methods:

• Synth-CLIP fine-tunes pre-trained CLIP models with tailored, domain-specific and shared prompts for visual samples.
• Integrates real and synthetic data, reorganizing visual features into the semantic space to expand data pools and diversity.
• Introduces a cross-domain feature alignment loss to match real and synthetic samples in the feature embedding space based on semantic consistency.

Main Results:

• Synth-CLIP effectively expands the data pool and enriches category diversity.
• Aligning visual and semantic distributions with synthetic data helps rebalance decision boundaries, even without novel visual samples.
• Experimental results show competitive performance across benchmarks, outperforming PromptSRC by 3.0% on novel classes across 11 datasets in open-vocabulary scenarios.

Conclusions:

• Synth-CLIP offers a robust method for improving VLM generalization, particularly in low-data scenarios for novel classes.
• The use of synthetic data and cross-domain alignment is crucial for enhancing model adaptability and performance.
• This approach demonstrates significant potential for open-vocabulary recognition and few-shot learning tasks.