A multi-stage dynamical fusion network for multimodal emotion recognition.

Area of Science:

• Affective computing
• Biomedical signal processing
• Machine learning

Background:

• Emotion recognition from physiological signals is a growing research area.
• Multimodal signals offer richer data than single modalities for emotion detection.
• Existing multimodal methods often neglect crucial cross-modal interactions by using one-stage fusion.

Purpose of the Study:

• To propose a novel multi-stage multimodal dynamical fusion network (MSMDFN) for emotion recognition.
• To address the limitations of one-stage fusion methods by incorporating cross-modal interactions.
• To achieve more accurate emotion recognition through fine-grained analysis of intermodal correlations.

Main Methods:

• Developed a multi-stage multimodal dynamical fusion network (MSMDFN).
• Explored latent interactions among features extracted from multiple physiological signal modalities.
• Implemented a multi-stage fusion process to capture unimodal, bimodal, and trimodal intercorrelations.

Main Results:

• The MSMDFN effectively obtains joint representations based on cross-modal correlations.
• The proposed method demonstrated superior performance compared to one-stage multimodal emotion recognition techniques.
• Experiments conducted on the DEAP multimodal benchmark dataset validated the effectiveness of MSMDFN.

Conclusions:

• The multi-stage fusion approach significantly enhances emotion recognition accuracy by leveraging cross-modal interactions.
• MSMDFN provides a more comprehensive analysis of multimodal physiological signals for emotion detection.
• This work advances the field of multimodal emotion recognition by introducing a dynamic, multi-stage fusion strategy.