TF-BERT: Tensor-based fusion BERT for multimodal sentiment analysis

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Unimodal sentiment analysis struggles with real-world complexity.
• Existing Transformer models for multimodal sentiment analysis are limited to processing only two modalities simultaneously.
• This limitation leads to insufficient information exchange and potential loss of emotional data.

Purpose of the Study:

• To propose a novel Tensor-based Fusion BERT (TF-BERT) model to address the limitations of traditional Crossmodal Transformer models.
• To enhance information exchange and emotional data representation in multimodal sentiment analysis.
• To enable simultaneous processing of three modalities for more comprehensive analysis.

Main Methods:

• Developed the Tensor-based Crossmodal Fusion (TCF) module integrated into BERT.
• Introduced the Tensor-based Crossmodal Transformer (TCT) module for simultaneous three-modality processing.
• Embedded TCF into multiple layers of BERT's Transformer for progressive, dynamic modality complementation.

Main Results:

• TF-BERT achieved state-of-the-art results on the CMU-MOSI and CMU-MOSEI datasets across most metrics.
• Ablation studies confirmed the effectiveness of both the TCF and TCT modules.
• The model demonstrated superior performance in progressively integrating and capturing complex emotional interactions across all modalities.

Conclusions:

• TF-BERT effectively overcomes the limitations of traditional models in multimodal sentiment analysis.
• The proposed TCF and TCT modules significantly improve information exchange and emotional representation.
• TF-BERT offers a more robust and comprehensive approach to analyzing complex emotional interactions in multimodal data.