Multimodal Composition Example Mining for Composed Query Image Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Composed query image retrieval involves finding a target image using a query composed of a reference image and a text description of modifications.
• Existing methods often fail to fully leverage multimodal characteristics, leading to suboptimal embedding spaces and inefficient training data utilization.
• Current learning objectives typically use query-level negatives, neglecting the nuanced multimodal nature of composed queries.

Purpose of the Study:

• To improve the learning objective for composed query image retrieval by focusing on multimodal fusion.
• To enhance the construction of the metric space by mining and generating effective hard negative examples.
• To address the limitations of conventional negative sampling in multimodal embedding learning.

Main Methods:

• Proposing a novel learning objective that constructs and mines hard negative examples from a multimodal fusion perspective.
• Creating component-level negative examples by pairing reference images with logically unpaired sentences.
• Introducing a new sentence augmentation technique to generate indistinguishable multimodal negative examples at the element level.

Main Results:

• The proposed method significantly improves the effectiveness of composed query image retrieval.
• Mining hard negative examples from component-level and element-level perspectives leads to better metric space optimization.
• Experiments on four real-world datasets validate the superior performance of the proposed approach compared to existing methods.

Conclusions:

• The developed method effectively addresses the limitations of current approaches in composed query image retrieval.
• Generating sophisticated negative examples is crucial for learning robust multimodal embedding spaces.
• This work offers a promising direction for advancing multimodal retrieval tasks through improved learning objectives.