Deep Multimodal Complementarity Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Complementarity is crucial for synergistic effects in complex data objects.
• Learning complementarity in multimodal data presents challenges in representation and metric definition.
• Existing similarity-based metrics fail to adequately capture complementarity.

Purpose of the Study:

• To propose a novel deep architecture for learning complementarity in multimodal multi-item data.
• To address the limitations of existing methods in measuring complementarity.
• To systematically extract both intramodal and intermodal complementarity.

Main Methods:

• A novel deep architecture with three modules: unimodal aggregation, cross-modal fusion, and interactive aggregation.
• Utilizing the TUBE distance metric for quantifying complementarity in the representation space.
• Developing methods to extract complementarity at both modality and item levels.

Main Results:

• The proposed model significantly outperforms state-of-the-art methods.
• Achieved +6.8% mean reciprocal rank (MRR) improvement in object classification.
• Achieved +3.0% MRR improvement in hold-out item prediction.
• Qualitative analysis confirms complementarity is distinct from similarity.

Conclusions:

• The novel deep architecture effectively learns complementarity in multimodal multi-item data.
• The TUBE distance metric provides a suitable measure for complementarity.
• The findings highlight the importance of complementarity beyond simple similarity.