Cross-modal credibility modelling for EEG-based multimodal emotion recognition.

Area of Science:

• Affective computing
• Multimodal machine learning
• Biomedical signal processing

Background:

• Emotion recognition using electroencephalography (EEG) is a growing field.
• Integrating EEG with peripheral physiological signals can improve recognition performance.
• Existing methods face challenges with modality heterogeneity and fusion credibility.

Purpose of the Study:

• To develop a novel multimodal physiological signal fusion model for credible EEG-based emotion recognition.
• To address modality heterogeneity and fusion credibility issues in multimodal emotion recognition.

Main Methods:

• Implemented a local self-attention transformer for intra-modal feature extraction.
• Devised a pairwise cross-attention transformer to capture inter-modal correlations.
• Introduced sequential pattern consistency to measure and ensure fusion credibility.

Main Results:

• Achieved accuracy improvements of 4.58% (DEAP) and 3.97% (MAHNOB-HCI) over state-of-the-art baselines.
• Improved F1 scores by 0.63% (DEAP) and 4.21% (MAHNOB-HCI).
• Demonstrated the effectiveness of credibility modeling through extensive experiments and analysis.

Conclusions:

• The proposed multimodal fusion architecture significantly enhances emotion recognition performance.
• Credibility modeling is essential for robust and reliable multimodal emotion recognition.
• The approach effectively handles modality heterogeneity and ensures fusion credibility.