Cross-modal dual subspace learning with adversarial network.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval is crucial for handling diverse data types.
• Key challenges include leveraging complementary data and minimizing heterogeneity.
• Existing methods struggle to fully bridge the gap between different modalities.

Purpose of the Study:

• To propose a novel network model, Dual Subspace learning with Adversarial Network (DSAN), for effective cross-modal retrieval.
• To address the challenges of utilizing complementary information and reducing the heterogeneous gap.
• To enhance the performance of cross-modal retrieval systems.

Main Methods:

• Developed dual subspaces (visual and textual) to extract modality-specific and underlying structural information.
• Introduced an improved quadruplet loss function considering both relative and absolute distances with hard sample mining.
• Proposed an intra-modal constrained loss to maximize distances between cross-modal positive and negative samples.
• Employed adversarial learning with feature preserving and modality classification as antagonists to narrow the heterogeneous gap.

Main Results:

• DSAN demonstrated superior performance compared to 9 state-of-the-art methods.
• Experiments were conducted on four diverse cross-modal datasets.
• The proposed dual subspace approach effectively mined underlying structure and modality-specific information.

Conclusions:

• DSAN successfully narrows the heterogeneous gap between different modalities.
• The model significantly outperforms existing methods in cross-modal retrieval tasks.
• The dual subspace learning and adversarial network approach offers a promising direction for multimodal data research.