Deep Hierarchical Multimodal Metric Learning.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Multimodal metric learning integrates heterogeneous data into a common subspace for cross-modal similarity computation.
• Existing methods often overlook label hierarchies, limiting performance on complex datasets with intercategory correlations.

Purpose of the Study:

• To develop a novel metric learning method for hierarchical labeled multimodal data.
• To address the limitations of non-hierarchical approaches in exploiting label structure.

Main Methods:

• Proposed deep hierarchical multimodal metric learning (DHMML) with layer-specific networks for each label hierarchy layer.
• Introduced a multilayer classification mechanism to preserve intra-layer semantics and inter-layer correlations.
• Integrated an adversarial learning mechanism to minimize the cross-modality gap, creating modality-invariant features.

Main Results:

• DHMML learns multilayer representations capturing hierarchical semantic similarities and intercategory relationships.
• The method generates discriminative and modality-invariant features for multimodal data.
• Experimental results on benchmark datasets demonstrate DHMML's superiority over state-of-the-art methods.

Conclusions:

• DHMML effectively addresses the challenge of metric learning with hierarchical labels in multimodal data.
• The proposed approach enhances the performance of cross-modal similarity computation by leveraging label hierarchy.
• DHMML offers a robust solution for analyzing complex, hierarchically structured multimodal datasets.