Multimodal prototypical network for interpretable sentiment classification

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Sentiment analysis increasingly uses multimodal video data (visual, acoustic, textual).
• Limited understanding exists on temporal segment contributions to model decisions in multimodal sentiment analysis.
• Existing interpretable methods struggle with multimodal interactions and temporal dependencies in video.

Purpose of the Study:

• To extend prototype-based interpretability to multimodal sentiment classification.
• To develop a method that identifies temporal segment contributions and modality-level feature importance.
• To improve the explainability of multimodal sentiment analysis models.

Main Methods:

• Proposed MultiModal Prototypical Networks (MMPNet) for multimodal sentiment classification.
• Extended prototype-based interpretability to handle multimodal video data.
• Developed techniques to identify time-level feature contributions and modality-level importance.

Main Results:

• MMPNet achieved superior performance, outperforming existing methods by 2.9% on CMU-MOSI and 1.6% on CMU-MOSEI.
• Demonstrated improved accuracy in multimodal sentiment classification tasks.
• Provided enhanced interpretability for model predictions.

Conclusions:

• MMPNet offers a novel approach to interpretable multimodal sentiment analysis.
• The method effectively explains predictions by analyzing temporal and modality features.
• MMPNet sets a new benchmark for accuracy and interpretability in video-based sentiment analysis.