Deep Multimodal Transfer Learning for Cross-Modal Retrieval.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Cross-modal retrieval (CMR) leverages multimedia data but requires extensive labeled datasets.
• Manual annotation is costly and time-consuming, hindering CMR model development.
• Existing transfer learning methods often assume identical label sets between source and target domains.

Purpose of the Study:

• To develop a deep multimodal transfer learning (DMTL) approach for effective knowledge transfer in CMR.
• To enable knowledge transfer from labeled source domains to unlabeled target domains with potentially disjoint label sets.
• To improve retrieval performance in scenarios with limited or no labeled data in the target domain.

Main Methods:

• Proposed a deep multimodal transfer learning (DMTL) approach for cross-modal retrieval (CMR).
• Employed a joint learning paradigm with iteratively updated pseudolabels for self-supervised knowledge transfer.
• Utilized multiple modality-specific neural networks to learn a shared semantic space, enforcing compactness of similar instances and scatter of dissimilar ones.
• Addressed the challenge of different or disjoint label sets between source and target domains.

Main Results:

• The proposed DMTL method demonstrated significant effectiveness in multimodal transfer learning.
• Achieved superior cross-modal retrieval performance compared to 11 state-of-the-art methods on four benchmarks.
• Validated the approach's ability to transfer knowledge across domains with differing label sets.

Conclusions:

• The DMTL approach offers a powerful solution for knowledge transfer in cross-modal retrieval, especially in low-resource or zero-shot scenarios.
• The method successfully bridges the domain gap and handles differing label spaces, outperforming existing techniques.
• This work advances the field of multimodal learning by enabling more flexible and efficient knowledge transfer for multimedia data retrieval.