Multimodal Fusion of Behavioral and Physiological Signals for Enhanced Emotion Recognition via Feature Decoupling and...

Area of Science:

• Affective computing
• Human-computer interaction
• Machine learning

Background:

• Multimodal emotion recognition integrates physiological and behavioral signals to understand complex human affective states.
• Existing methods face challenges with feature redundancy, modality heterogeneity, and limited cross-modal supervision.

Purpose of the Study:

• To propose a novel Multimodal Disentangled Knowledge Distillation framework for enhanced emotion recognition.
• To explicitly disentangle modality-shared and modality-specific features.
• To improve cross-modal knowledge transfer using a graph-based distillation module.

Main Methods:

• A dual-stream representation learning architecture separates common and unique feature subspaces across modalities.
• A directed and learnable modality graph models semantic transfer strength between modalities.
• The framework was validated on MAHNOB-HCI and DEAP datasets for regression and classification tasks.

Main Results:

• The proposed method achieved state-of-the-art performance on benchmark datasets.
• Statistical significance was confirmed using paired two-tailed t-tests.
• Qualitative analysis demonstrated the effectiveness of feature disentanglement and dynamic knowledge transfer.

Conclusions:

• The framework offers a unified, interpretable, and robust approach to multimodal emotion understanding.
• This work advances affective computing for real-world human-machine interaction scenarios.
• The method effectively addresses feature redundancy and modality heterogeneity in multimodal emotion recognition.