Hypergraph multi-modal learning for EEG-based emotion recognition in conversation

Emotion Recognition in Conversation (ERC) is valuable for diagnosing health conditions such as autism and depression (Maryenko, 2024), and for understanding the emotions of individuals who struggle to express their feelings. Current ERC methods primarily rely on semantic, audio and video data but face significant challenges in integrating physiological signals such as Electroencephalography (EEG), which has low signal-to-noise ratios, inter-subject variability, and temporal alignment issues. This research proposes Hypergraph Multi-Modal Learning (Hyper-MML), a novel framework for identifying emotions in conversation. Hyper-MML effectively integrates EEG with audio and video information to capture complex emotional dynamics. Firstly, we introduce an Adaptive Brain Encoder with Mutual-cross Attention (ABEMA) module for processing EEG signals. This module captures emotion-relevant features across different frequency bands and adapts to subject-specific variations through hierarchical mutual-cross attention mechanisms. Secondly, we propose an Adaptive Hypergraph Fusion Module (AHFM) to actively model the higher-order relationships among multi-modal signals in ERC. Experimental results on the EAV and AFFEC datasets demonstrate that our Hyper-MML model significantly outperforms current state-of-the-art methods. The proposed Hyper-MML can serve as an effective communication tool for healthcare professionals, enabling better engagement with patients who have difficulty expressing their emotions. The official implementation codes are available at https://github.com/NZWANG/Hyper-MML.