Decoupled Hierarchical Distillation for Multimodal Emotion Recognition

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing
• Speech Processing

Background:

• Human multimodal emotion recognition (MER) integrates language, visual, and acoustic data to infer emotions.
• Existing MER methods face challenges with multimodal heterogeneity and varying modality contributions.
• Addressing these limitations is crucial for advancing accurate emotion AI.

Purpose of the Study:

• To propose a novel framework, Decoupled Hierarchical Multimodal Distillation (DHMD), for enhanced MER.
• To decouple modality features into irrelevant and exclusive components for better representation.
• To improve cross-modal feature alignment and recognition accuracy.

Main Methods:

• Utilized a self-regression mechanism to decouple modality features.
• Employed a two-stage knowledge distillation (KD) strategy: coarse-grained (Graph Distillation Unit) and fine-grained (dictionary matching).
• Implemented a dynamic graph for adaptive inter-modal distillation and dictionary matching for semantic alignment.

Main Results:

• DHMD achieved significant relative improvements: 1.3%/2.4% (ACC7), 1.3%/1.9% (ACC2), and 1.9%/1.8% (F1) on CMU-MOSI/CMU-MOSEI datasets.
• The framework consistently outperformed state-of-the-art MER methods.
• Visualization confirmed meaningful distribution patterns in decoupled features and distillation mechanisms.

Conclusions:

• DHMD effectively addresses multimodal heterogeneity in MER.
• The proposed hierarchical distillation strategy enhances cross-modal alignment and recognition performance.
• DHMD represents a significant advancement in the field of emotion recognition technology.