Related Experiment Video
Updated: Nov 6, 2025
12:55
Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties
Published on: September 27, 2020
8.7K
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis
Yao-Hung Hubert Tsai1, Martin Q Ma1, Muqiao Yang1
1Carnegie Mellon University, Pittsburgh, PA, USA.
Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing
|May 10, 2021
Summary
This study introduces Multimodal Routing, a new method for interpreting human language across different information sources. It enhances understanding of how various communication styles influence predictions, offering both global and local insights.
Area of Science:
- Artificial Intelligence
- Natural Language Processing
- Machine Learning
Background:
- Human language relies on multiple information sources (modalities) like tone, facial expressions, and speech.
- Current multimodal learning models excel in tasks like sentiment analysis but often lack interpretability, acting as black boxes.
Purpose of the Study:
- To develop a novel method, Multimodal Routing, for enhancing the interpretability of multimodal learning systems.
- To dynamically adjust the influence of different input modalities based on individual data samples.
Main Methods:
- Propose Multimodal Routing, a technique that assigns dynamic weights to input modalities and output representations.
- The routing mechanism identifies the importance of individual modalities and cross-modality interactions.
Main Results:
- Multimodal Routing provides interpretable insights into modality-prediction relationships.
- Interpretations are available globally (dataset-wide trends) and locally (per-sample analysis).
- The method achieves performance competitive with state-of-the-art approaches.
Conclusions:
- Multimodal Routing offers a significant advancement in understanding how different communication modalities contribute to predictions.
- This approach enhances transparency in complex AI systems, enabling more reliable human-centric task performance.
