Geometry Sensitive Cross-Modal Reasoning for Composed Query Based Image Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Composed Query Based Image Retrieval (CQBIR) involves retrieving images based on a reference image and a textual modification.
• Existing CQBIR methods struggle with the semantic gap between image and text, often failing to model query interactions or spatial-visual-semantic relationships.

Purpose of the Study:

• To propose a geometry-sensitive cross-modal reasoning network to address the challenges in CQBIR.
• To effectively model the geometric information and visual-semantic relationships within composed queries.

Main Methods:

• Developed a geometry-sensitive inter-modal attention module (GS-IMA) to incorporate spatial structure into attention mechanisms.
• Introduced a text-guided visual reasoning module (TG-VR) to handle semantics not present in the reference image.
• Jointly modeled geometric information and visual-semantic relationships for effective feature learning.

Main Results:

• The proposed network learns effective features for composed queries, even without literal alignment.
• Achieved favorable performance compared to state-of-the-art methods on three standard benchmarks.
• Demonstrated the efficacy of jointly modeling geometric and visual-semantic information.

Conclusions:

• The proposed geometry-sensitive cross-modal reasoning network offers a significant advancement in CQBIR.
• The method successfully bridges the semantic gap by considering spatial structure and visual-semantic relationships.
• This approach provides a more robust solution for complex image retrieval tasks.