Integrating Multimodal Information in Large Pretrained Transformers.

Area of Science:

• Natural Language Processing (NLP)
• Multimodal Machine Learning
• Computer Vision and Acoustics

Background:

• Transformer-based models (BERT, XLNet) excel in NLP tasks through fine-tuning.
• Existing models struggle with multimodal data (vision, acoustics) in face-to-face communication.
• Pre-trained models lack mechanisms for integrating non-language modalities.

Purpose of the Study:

• To propose a novel method for adapting pre-trained Transformer models to multimodal data.
• To enhance sentiment analysis performance using visual and acoustic information.
• To enable Transformer models to process nonverbal cues in communication.

Main Methods:

• Introduced the Multimodal Adaptation Gate (MAG) as an attachment to BERT and XLNet.
• MAG conditions internal representations on visual and acoustic modalities during fine-tuning.
• Evaluated MAG-BERT and MAG-XLNet on CMU-MOSI and CMU-MOSEI multimodal sentiment analysis datasets.

Main Results:

• MAG significantly boosts sentiment analysis performance compared to language-only models.
• MAG-BERT and MAG-XLNet outperform previous multimodal sentiment analysis baselines.
• MAG-XLNet achieved human-level performance on the CMU-MOSI dataset for the first time.

Conclusions:

• The Multimodal Adaptation Gate (MAG) effectively integrates nonverbal data into Transformer models.
• MAG enables state-of-the-art multimodal sentiment analysis, reaching human-level performance.
• This approach advances NLP for modeling complex, face-to-face communication.