External Guidance Incomplete Cross-Modal Hashing

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal hashing (CMH) methods assume complete, paired multimodal data, which is often not the case in real-world scenarios.
• Existing incomplete CMH methods struggle with missing modalities due to sensitivity to distributional shifts and reliance on internal data signals.

Purpose of the Study:

• To propose a novel framework, External Guidance Incomplete Cross-modal Hashing (EGICH), to address limitations in existing incomplete CMH methods.
• To leverage external knowledge bases for more robust reconstruction of missing modalities and mitigate cross-modal bias.

Main Methods:

• Developed a Completion with External Guidance (CEG) module to utilize external knowledge for accurate semantic reconstruction of missing samples.
• Introduced a Consistency Learning with External Guidance (CLEG) module to align representations with label semantics using externally guided features.
• Implemented a Semantic-aware Contrastive Hashing (SCH) module to refine feature distribution based on semantic similarity for improved discrimination.

Main Results:

• EGICH consistently and significantly outperforms 11 state-of-the-art methods.
• The framework demonstrates robust performance across various modality-missing scenarios.
• External knowledge integration proves effective in enhancing incomplete cross-modal hashing.

Conclusions:

• EGICH is the first framework to incorporate external knowledge into incomplete cross-modal hashing.
• The proposed method effectively handles missing modalities by leveraging external semantic information.
• EGICH offers a significant advancement in cross-modal retrieval with incomplete data.