Unpaired Image-Text Matching via Multimodal Aligned Conceptual Knowledge

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Current image-text matching models rely on large paired datasets for supervised learning.
• Human multimodal knowledge allows image-text matching without explicit paired data.
• A gap exists in learning image-text associations from unpaired data.

Purpose of the Study:

• To propose a novel method for unpaired image-text matching.
• To develop a system that learns multimodal knowledge without paired image-text examples.
• To enhance the performance of existing image-text matching models in zero-shot and cross-dataset scenarios.

Main Methods:

• Propose Multimodal Aligned Conceptual Knowledge (MACK) framework.
• Pretrain general multimodal knowledge using word-region associations.
• Refine knowledge using self-supervised learning on unpaired image-text data.
• Compute image-text similarity via region-word similarity aggregation.

Main Results:

• MACK effectively performs unpaired image-text matching.
• The method refines general knowledge into domain-specific knowledge.
• MACK can be integrated as a re-ranking method to boost existing model performance.
• Significant improvements observed in zero-shot and cross-dataset matching tasks.

Conclusions:

• MACK offers a viable solution for image-text matching with unpaired data.
• The approach demonstrates the potential of leveraging conceptual knowledge for multimodal tasks.
• MACK provides a complementary method to enhance current state-of-the-art models.