M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis.

Area of Science:

• Natural Language Processing
• Human-Computer Interaction
• Artificial Intelligence

Background:

• Multimodal sentiment analysis leverages text, voice, and facial expressions to understand human attitudes.
• Current deep learning models for this task are often "black boxes," lacking transparency in their decision-making processes.
• Existing explainability techniques primarily focus on unimodal data, leaving a gap in understanding multimodal model behavior.

Purpose of the Study:

• To introduce M2 Lens, an interactive visual analytics system designed to explain multimodal sentiment analysis models.
• To provide insights into the intra- and inter-modal interactions influencing sentiment predictions.
• To enhance the transparency and interpretability of complex multimodal AI systems.

Main Methods:

• Development of an interactive visual analytics system, M2 Lens.
• Visualization of intra- and inter-modal interactions at global, subset, and local levels.
• Identification of feature influences and interaction types (dominance, complement, conflict) within multimodal models.

Main Results:

• M2 Lens effectively visualizes and explains how multimodal models utilize different communication channels for sentiment analysis.
• The system summarizes the impact of key interaction types on model predictions.
• Case studies and expert feedback confirm M2 Lens's ability to provide deep insights into model behavior.

Conclusions:

• M2 Lens offers a novel approach to understanding and interpreting multimodal sentiment analysis models.
• The system facilitates deeper insights into the complex interplay of linguistic, acoustic, and visual features.
• Enhanced model explainability can foster greater trust and further development in multimodal AI research.