Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning.

Area of Science:

• Computer Science
• Neuroscience
• Affective Computing

Background:

• Multimodal emotion recognition leverages diverse data sources like physiological signals, visual cues, and audio-visual content.
• Existing methods face challenges in managing redundant or conflicting cross-modal information and capturing implicit correlations.
• Effective integration of disparate data streams is crucial for accurate human emotion understanding.

Purpose of the Study:

• To develop a novel multimodal emotion recognition framework integrating audio-visual features with electroencephalography (EEG) data.
• To enhance emotion classification accuracy by effectively processing and fusing information from multiple modalities.
• To address limitations in current methods regarding redundant/conflicting information and inter-modal correlations.

Main Methods:

• Utilized modality-specific encoders for extracting spatiotemporal features from audio-visual and EEG data.
• Employed contrastive learning to align features and capture inter-modal relationships.
• Incorporated cross-modal attention mechanisms for effective feature fusion.

Main Results:

• The proposed framework demonstrated high effectiveness in emotion recognition across multiple datasets.
• Integration of audio-visual features with EEG data significantly improved classification accuracy.
• The approach successfully captured inter-modal relationships and fused features effectively.

Conclusions:

• The novel multimodal framework integrating audio-visual and EEG data offers a significant advancement in emotion recognition.
• The method provides a robust solution for handling complex emotional states by leveraging complementary information from different modalities.
• This research highlights the potential of combining neurophysiological data with behavioral cues for more accurate and comprehensive emotion understanding.