Dynamic Bit-Wise Semantic Transformer Hashing for Multi-Modal Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Multi-modal hashing encodes diverse data into binary codes for efficient retrieval.
• Existing methods struggle with modality gaps, bit independence, and capturing fine-grained semantics.

Purpose of the Study:

• To introduce a novel framework, Dynamic Bit-wise Semantic Transformer Hashing (DBSTH), to address limitations in current multi-modal hashing techniques.
• To improve the semantic representation capacity and retrieval efficiency of multi-modal data.

Main Methods:

• DBSTH treats each hash bit as a semantic concept for modality alignment and fusion.
• Employs a dynamic unit fusion strategy and a transformer encoder for concept refinement.
• Incorporates label prototype learning and masked concept learning for enhanced concept acquisition and robustness.

Main Results:

• DBSTH effectively bridges the heterogeneous modality gap at the concept level.
• Achieves superior performance in conventional, noisy, and open-set multi-modal retrieval scenarios.
• Demonstrates enhanced semantic representation and bit independence.

Conclusions:

• DBSTH offers a robust and effective solution for multi-modal hashing and retrieval.
• The concept-level alignment and learning strategies significantly improve fine-grained semantic correlation capture.
• The framework shows promise for efficient and accurate multimedia retrieval across various conditions.