Joint Feature Synthesis and Embedding: Adversarial Cross-Modal Retrieval Revisited

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Generative Adversarial Networks (GANs) excel at modeling data distributions.
• Cross-modal GANs are a hotspot for learning compatible features across modalities.
• Existing cross-modal GANs suffer from high costs for labeled data, unstable training, and limited extensibility.

Purpose of the Study:

• To propose Joint Feature Synthesis and Embedding (JFSE), a novel method to address shortcomings in current cross-modal GANs.
• To enable joint multimodal feature synthesis and common embedding space learning.
• To improve cross-modal retrieval accuracy, especially for zero-shot and generalized zero-shot scenarios.

Main Methods:

• JFSE utilizes two coupled conditional Wasserstein GAN modules for synthesizing meaningful, correlated multimodal features guided by class label word embeddings.
• Employs three advanced distribution alignment schemes with cycle-consistency constraints to maintain semantic compatibility.
• Facilitates knowledge transfer in a common embedding space for both real and synthetic cross-modal features.

Main Results:

• JFSE learns a more effective common embedding space, capturing cross-modal correlations.
• The method demonstrates enhanced knowledge transfer capabilities for new classes.
• Achieves significant accuracy improvements on standard, zero-shot, and generalized zero-shot retrieval tasks across four datasets compared to over ten state-of-the-art methods.

Conclusions:

• JFSE offers a robust solution to limitations in existing cross-modal GANs.
• The proposed method enhances cross-modal feature learning and retrieval performance.
• JFSE shows strong potential for applications requiring effective cross-modal understanding and retrieval, including novel class scenarios.