Clustering swap prediction for image-text pre-training.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal model pre-training significantly impacts downstream task performance.
• Clustering learning offers benefits but faces challenges with open image-text pairs.
• Existing methods struggle with the scale and openness of web-scale alt-text data.

Purpose of the Study:

• To propose a novel approach for learning image-text clustering embedding space.
• To address the challenges of multimodal clustering with open image-text pairs.
• To develop a method that allows for an open number of clusters for web-scale data.

Main Methods:

• Introduced a clustering swap prediction strategy for interaction prediction between image and text features.
• Employed a distillation learning approach for efficient training of image and text encoders.
• Pre-trained the model end-to-end using large-scale image-text pairs with text and image as ground truth for swap prediction.

Main Results:

• Achieved state-of-the-art performance on multiple downstream fine-tuning and zero-shot tasks.
• Demonstrated effective representation learning through swap prediction.
• Evaluated the image-encoder's performance on downstream visual tasks.

Conclusions:

• The proposed Clus method effectively learns image-text clustering embeddings.
• Clus offers a scalable solution for multimodal clustering with open-world data.
• The approach shows strong generalization capabilities across diverse vision-language tasks.