Global and Local Semantic Completion Learning for Vision-Language Pre-Training.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Cross-modal alignment is vital for vision-language pre-training (VLP) models.
• Existing masked modeling tasks in VLP primarily focus on local-local alignment, neglecting global semantic features.
• This limitation hinders the alignment of global representations with local features from other modalities.

Purpose of the Study:

• To propose a novel Global and Local Semantic Completion Learning (GLSCL) task for VLP.
• To simultaneously facilitate global-local and local-local alignment.
• To enhance the cross-modal alignment capabilities of VLP models.

Main Methods:

• Introduced the GLSCL task, comprising masked global semantic completion (MGSC) and masked local token completion (MLTC).
• MGSC focuses on learning representative global features, while MLTC reconstructs modal-fusion local tokens.
• Developed a flexible vision encoder for simultaneous image-text and video-text tasks and introduced the ALIGN-BENCH validation benchmark.

Main Results:

• The proposed GLSCL task effectively complements missing semantics and recovers global and local features through cross-modal interactions.
• MGSC enhances global feature representation, impacting downstream task performance.
• MLTC improves the comprehension of multimodal data by reconstructing local tokens.

Conclusions:

• GLSCL significantly improves cross-modal alignment by addressing both global and local feature learning.
• The method achieves state-of-the-art performance on diverse vision-language benchmarks, including visual question answering and image-text retrieval.
• The flexible vision encoder and ALIGN-BENCH contribute to evaluating and advancing VLP models.