Mamba-Based Enhanced Multimodal Emotion Recognition with EEG Guidance.

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Multimodal emotion recognition combines data from various sources for better accuracy.
• Existing methods struggle with irrelevant or conflicting information across modalities.
• Mamba's efficiency in filtering data and modeling long-range dependencies inspires new approaches.

Purpose of the Study:

• To propose MambaMER, a Mamba-based paradigm for enhanced EEG-guided multimodal emotion recognition.
• To address performance limitations caused by cross-modal information conflicts.
• To improve the accuracy and efficiency of emotion recognition systems.

Main Methods:

• Developed a multi-scale EEG-guided conflict suppression module using a selective cross state space model.
• Engineered a novel cross-modal fusion mechanism (Mutual-Cross-Mamba and Merge-Mamba).
• Utilized Mamba's selective mechanism for dynamic parameter adjustment to filter irrelevant data.

Main Results:

• The proposed method significantly suppresses irrelevant or conflicting information.
• Achieved enhanced integration of complementary features between EEG and eye movement modalities.
• Demonstrated superior performance over state-of-the-art methods on SEED, SEED-IV, and SEED-V datasets.

Conclusions:

• MambaMER effectively overcomes cross-modal conflicts in multimodal emotion recognition.
• The approach enhances recognition accuracy and maintains linear computational complexity.
• This paradigm offers a promising direction for advanced emotion recognition research.