Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Composed Image Retrieval (CIR) involves finding target images from multimodal queries (reference image + text).
• Existing CIR methods often overlook multiple matching factors and underutilize unlabeled data, limiting performance.
• Challenges include modeling latent matching factors without supervision and leveraging unlabeled pairs for better generalization.

Purpose of the Study:

• To address limitations in CIR by developing a model that accounts for multiple matching factors and unlabeled data.
• To propose a novel network architecture and training paradigm for enhanced CIR performance.
• To improve the generalization ability of CIR models.

Main Methods:

• Proposed a CLIP-Transformer based muLtI-factor Matching Network (LIMN) with modules for latent factor mining, matching token learning, and query-target matching.
• Introduced an iterative dual self-training paradigm (LIMN+) to leverage unlabeled reference-target image pairs in a weakly-supervised manner.
• Evaluated performance on FashionIQ, Shoes, CIRR, and Fashion200 K datasets.

Main Results:

• LIMN effectively models multiple matching factors in a latent way.
• LIMN+ significantly enhances CIR performance by utilizing unlabeled data through self-training.
• Both LIMN and LIMN+ demonstrate substantial improvements over state-of-the-art baselines across multiple datasets.

Conclusions:

• The proposed LIMN and LIMN+ models offer a significant advancement in composed image retrieval.
• Effectively modeling multiple factors and utilizing unlabeled data are crucial for improving CIR.
• The developed methods show strong potential for real-world applications requiring nuanced image retrieval.