MHTN: Modal-Adversarial Hybrid Transfer Network for Cross-Modal Retrieval

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Cross-modal retrieval enables searching across diverse data types like text, images, and audio.
• Existing deep learning methods struggle with insufficient cross-modal training data, leading to overfitting.
• Current transfer learning primarily transfers knowledge between single-modal datasets, neglecting shared semantic knowledge.

Purpose of the Study:

• To propose a novel network for effective knowledge transfer from single-modal to cross-modal domains.
• To learn a shared cross-modal common representation despite limited labeled cross-modal data.
• To leverage abundant single-modal data for boosting cross-modal retrieval performance.

Main Methods:

• Developed a modal-adversarial hybrid transfer network (MHTN) with two subnetworks.
• Implemented a modal-sharing knowledge transfer subnetwork using a star network structure for knowledge distillation.
• Employed a modal-adversarial semantic learning subnetwork with adversarial training for enhanced representation.

Main Results:

• The proposed MHTN effectively transfers knowledge from single-modal to cross-modal domains.
• The network learns a discriminative yet modality-indiscriminative common representation.
• Experiments on four datasets demonstrated the superior effectiveness of the MHTN approach.

Conclusions:

• MHTN successfully addresses the challenge of insufficient cross-modal training data.
• The approach effectively exploits knowledge from single-modal datasets for cross-modal retrieval.
• The proposed method enhances cross-modal semantic consistency and retrieval accuracy.