A novel signal channel attention network for multi-modal emotion recognition

Area of Science:

• Affective computing
• Biomedical signal processing

Background:

• Physiological signal recognition is key for emotion recognition.
• Current multi-modal fusion models struggle with complex signals due to fusion methods and attention mechanisms, limiting performance.

Purpose of the Study:

• To propose a novel model, Signal Channel Attention Network (SCA-Net), to overcome limitations in multi-modal emotion recognition.
• To enhance classification performance by developing and integrating effective attention mechanisms.

Main Methods:

• Developed SCA-Net with an encoder, attention fusion module, and decoder.
• Introduced five novel attention mechanisms within the fusion module.
• Conducted comparative and ablation experiments on the MAHNOB-HCI dataset.

Main Results:

• Attention mechanisms significantly improved accuracy and F1 scores compared to the baseline model.
• Ablation experiments confirmed the benefits of multi-modal fusion.
• Early stopping parameters were adjusted to prevent overfitting.

Conclusions:

• The proposed SCA-Net effectively enhances emotion recognition from physiological signals.
• Attention mechanisms and multi-modal fusion are crucial for improving classification performance in complex affective computing tasks.