Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Text-to-image person retrieval (TIPR) faces challenges due to modality heterogeneity between text descriptions and images.
• Existing methods often overlook fine-grained differences or require explicit part alignments.
• Current TIPR systems are predominantly English-centric, limiting their global applicability.

Purpose of the Study:

• To pioneer a multilingual TIPR task and establish a comprehensive benchmark.
• To develop a novel framework, Bi-IRRA, for effective cross-lingual and cross-modal alignment.
• To overcome the limitations of existing English-centric and alignment-focused TIPR approaches.

Main Methods:

• Leveraged large language models for initial translations and refined them with domain-specific knowledge to create a multilingual TIPR benchmark.
• Proposed Bi-IRRA: a Bidirectional Implicit Relation Reasoning and Aligning framework.
• Integrated a bidirectional implicit relation reasoning module for enhanced local relation modeling and a multi-dimensional global alignment module to bridge modality gaps.

Main Results:

• Achieved new state-of-the-art results across all developed multilingual TIPR datasets.
• Demonstrated the effectiveness of Bi-IRRA in handling modality heterogeneity and cross-lingual alignment.
• Successfully addressed the limitations of English-centric and less granular alignment strategies in prior works.

Conclusions:

• The proposed multilingual TIPR benchmark and Bi-IRRA framework represent a significant advancement in cross-modal retrieval.
• Bi-IRRA offers a robust solution for multilingual person identification by implicitly reasoning relations across modalities and languages.
• This work paves the way for more inclusive and effective person retrieval systems in diverse linguistic contexts.