Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Machine Learning

Background:

• Multimodal fusion analyzes spoken language with visual and prosodic cues.
• Current multimodal models lack clarity on interaction learning versus independent modality processing.

Purpose of the Study:

• To propose Multimodal Residual Optimization (MRO) for separating unimodal, bimodal, and trimodal interactions.
• To enhance the interpretability of multimodal models by quantifying interaction effects.

Main Methods:

• MRO prioritizes learning simpler unimodal contributions before complex bimodal and trimodal interactions.
• Bimodal predictions are trained to correct unimodal prediction residuals, focusing on remaining interactions.

Main Results:

• MRO effectively separates unimodal, bimodal, and trimodal interactions.
• The proposed method maintains or improves predictive performance.
• A human perception study confirmed MRO's learned interactions align with human judgments.

Conclusions:

• MRO provides a quantifiable and interpretable approach to multimodal interaction analysis.
• The method enhances understanding of how different modalities contribute to overall performance.
• MRO offers a principled way to build more transparent and effective multimodal systems.