Dual Retrieval Queries Fine-Tuning for Composed Image Retrieval

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Composed Image Retrieval (CIR) is a multi-modal task requiring target image retrieval based on a reference image and modification text.
• Existing methods often suffer from modality redundancy, where one modality dominates the retrieval process, limiting performance.
• Effectively integrating visual and textual information is crucial for accurate CIR.

Purpose of the Study:

• To address the limitations of existing CIR methods, particularly modality redundancy.
• To propose a novel approach, DRQ-CIR, that fully leverages multi-modal information for enhanced retrieval.
• To improve the accuracy and effectiveness of composed image retrieval.

Main Methods:

• Proposed Dual Retrieval Queries Fine-Tuning for Composed Image Retrieval (DRQ-CIR).
• Introduced a Bilateral Multi-Modal Fusion (BMMF) module using pre-trained VLMs to create enriched retrieval queries.
• Developed a Dual Retrieval Queries Fine-Tuning (DRQ-FT) module with latent prompts for enhanced queries.
• Employed contrastive learning with dual retrieval queries and a bi-directional training paradigm.

Main Results:

• The proposed DRQ-CIR method demonstrated significant improvements in composed image retrieval.
• Effectiveness validated across four established benchmark datasets, showcasing robust performance.
• The asymmetric fusion mechanism successfully generated dual retrieval queries of different granularity, mitigating modality redundancy.

Conclusions:

• DRQ-CIR effectively overcomes modality redundancy by fully utilizing multi-modal information.
• The novel dual retrieval query generation and bi-directional training paradigm enhance CIR performance.
• The method offers a promising advancement for multi-modal retrieval tasks.