FAST-GOAL: Fast and Efficient Global-Local Object Alignment Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-language models (VLMs) like CLIP excel at image-text alignment but struggle with lengthy descriptions.
• Pre-training on short captions limits VLMs' ability to process detailed textual information.

Purpose of the Study:

• To introduce FAST-GOAL, an efficient fine-tuning method to improve CLIP's performance on long and detailed text descriptions.
• To enhance the global-local semantic alignment capabilities of vision-language models.

Main Methods:

• FAST-GOAL employs Fast Local Image-Sentence Matching (FLISM) using object detection and spatial division for region-sentence correspondence.
• Token Similarity-based Learning (TSL) maximizes similarity between image patch tokens and text embeddings for detailed feature alignment.
• Introduced GLIT100k dataset with global image-lengthy caption pairs and context-derived local pairs.

Main Results:

• FAST-GOAL significantly improves performance on long caption datasets (DOCCI, DCI) compared to baselines.
• The method demonstrates effectiveness in adapting CLIP for detailed textual descriptions.
• Maintained computational efficiency during fine-tuning.

Conclusions:

• FAST-GOAL effectively enhances vision-language models' ability to handle lengthy and detailed text.
• The global-local semantic alignment approach offers a promising direction for improving VLM performance.
• Efficient fine-tuning is crucial for adapting powerful pre-trained models to specific downstream tasks.