Multimodal Transformer for Unaligned Multimodal Language Sequences.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Human language is inherently multimodal, integrating natural language, facial gestures, and acoustic behaviors.
• Modeling multimodal time-series data faces challenges like data non-alignment and capturing long-range dependencies.

Purpose of the Study:

• To introduce a novel model, the Multimodal Transformer (MulT), for end-to-end processing of multimodal human language data.
• To address inherent data non-alignment and long-range dependency issues without explicit data alignment.

Main Methods:

• The proposed Multimodal Transformer (MulT) utilizes directional pairwise cross-modal attention.
• This attention mechanism facilitates interaction between multimodal sequences across different time steps.
• The model latently adapts data streams from one modality to another.

Main Results:

• MulT demonstrates superior performance on both aligned and non-aligned multimodal time-series datasets.
• The model significantly outperforms existing state-of-the-art methods.
• Empirical analysis confirms MulT's ability to capture correlated cross-modal signals.

Conclusions:

• The Multimodal Transformer (MulT) effectively models multimodal human language data.
• The directional pairwise cross-modal attention mechanism is key to MulT's success.
• MulT offers a robust solution for analyzing complex, real-world multimodal interactions.