X ²-VLM: All-in-One Pre-Trained Model for Vision-Language Tasks

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Existing vision-language pre-training methods primarily focus on image-text alignments.
• Some approaches utilize object detectors for object-level vision-language alignments.
• A gap exists in learning multi-grained alignments and localization simultaneously.

Purpose of the Study:

• To propose a unified pre-training framework for multi-grained vision-language alignments.
• To introduce X²-VLM, an all-in-one model unifying image-text and video-text pre-training.
• To enable learning of unlimited visual concepts with diverse text descriptions.

Main Methods:

• Developed a unified pre-training framework enabling simultaneous multi-grained aligning and multi-grained localization.
• Presented X²-VLM, a flexible, modular, all-in-one model architecture.
• Integrated both image-text and video-text pre-training within a single model.

Main Results:

• X²-VLM achieved state-of-the-art performance on both image-text and video-text tasks at base and large scales.
• The model demonstrated a favorable trade-off between performance and model scale.
• Showcased high transferability, outperforming multilingual models when using XLM-R without multilingual pre-training.

Conclusions:

• X²-VLM offers a powerful and versatile approach to vision-language pre-training.
• The modular design facilitates adaptation to various languages and domains.
• The unified framework advances the field of multi-modal learning.