Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Hypergraph multi-modal learning for EEG-based emotion recognition in conversation.

Zijian Kang1, Yueyang Li1, Shengyu Gong2

  • 1Lab of Digital Image and Intelligent Computation, Shanghai Maritime University, Shanghai, 201306, China; Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong Special Administrative Region, China.

Neural Networks : the Official Journal of the International Neural Network Society
|May 8, 2026
PubMed
Summary

Related Concept Videos

Labeling Emotion01:20

Labeling Emotion

Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

RND3 Enhances Cardiac Glucose Metabolism Through Inhibiting ACAT1-Dependent PDHA1 Acetylation and Protects Against Ischemia-Reperfusion Injury.

Circulation·2026
Same author

Pyroptosis: Novel Targets in Molecular Mechanisms and Drug Therapy Research for Myocardial Ischemia Reperfusion Injury.

Reviews in cardiovascular medicine·2026
Same author

From in vitro to in vivo: hypoxia attenuates replicative senescence and preserves therapeutic activity in long-term passaged human umbilical cord-derived mesenchymal stem cells.

Biological research·2026
Same author

Imagined Speech Brain-Computer Interface: A Task-Oriented Review of Neural Decoding.

Sensors (Basel, Switzerland)·2026
Same author

BnaA02.TOP1α and BnaC02.TOP1α redundantly regulate floral organ development in Brassica napus L.

Plant physiology·2026
Same author

A curved-beam piezoelectric MEMS resonator featuring multiple temperature plateaus with enhanced stability.

Microsystems & nanoengineering·2026
Same journal

Observer-based ADP for secure resource allocation in high-order nonlinear multi-agent systems under FDI attacks.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Concept mask-aware pruning and augmentation for few sample model compression.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Hindsight-based state space exploration via counterfactual intrinsic reward assignment.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Integrating visual and language cues via state space models for medical image segmentation.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

DNA: Improving text-based person search through distillation learning, negated relation-aware learning, and augmented representation learning.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

MCFusion-DDI: Multimodal cross-attention fusion of local-global features and latent drug associations for explainable DDI prediction.

Neural networks : the official journal of the International Neural Network Society·2026
See all related articles
This summary is machine-generated.

This study introduces Hypergraph Multi-Modal Learning (Hyper-MML) to improve emotion recognition in conversation by integrating Electroencephalography (EEG) with audio and video data. The novel framework significantly enhances accuracy, aiding communication for individuals with emotional expression difficulties.

Area of Science:

  • Neuroscience
  • Artificial Intelligence
  • Human-Computer Interaction

Background:

  • Emotion Recognition in Conversation (ERC) aids in diagnosing conditions like autism and depression.
  • Current ERC methods struggle to integrate physiological signals like Electroencephalography (EEG) due to noise and variability.
  • Effective integration of multi-modal data is crucial for advancing ERC.

Purpose of the Study:

  • To propose a novel framework, Hypergraph Multi-Modal Learning (Hyper-MML), for enhanced emotion recognition in conversation.
  • To effectively integrate Electroencephalography (EEG) signals with audio and video data for a comprehensive emotional analysis.
  • To address the challenges of low signal-to-noise ratios and inter-subject variability in EEG data.

Main Methods:

Keywords:
EEG-based emotion recognition (EER)Electroencephalography (EEG)Emotion recognition in conversation (ERC)Hypergraph learningMulti-modal fusionMutual-cross attention

Related Experiment Videos

  • Developed an Adaptive Brain Encoder with Mutual-cross Attention (ABEMA) module to process EEG signals, capturing emotion-relevant features across frequency bands.
  • Introduced an Adaptive Hypergraph Fusion Module (AHFM) to model higher-order relationships among multi-modal signals.
  • Utilized hierarchical mutual-cross attention mechanisms for subject-specific EEG adaptation.

    • Main Results:

    • The Hyper-MML framework demonstrated significant performance improvements over existing state-of-the-art methods on the EAV and AFFEC datasets.
    • Successfully integrated EEG data with audio and video, capturing complex emotional dynamics.
    • Validated the efficacy of ABEMA and AHFM modules in processing and fusing multi-modal emotional data.

    Conclusions:

    • Hyper-MML offers a robust solution for emotion recognition in conversation by effectively fusing multi-modal data, including EEG.
    • The framework has the potential to serve as a valuable communication tool in healthcare, particularly for patients with emotional expression challenges.
    • This research advances the field of multi-modal emotion recognition, paving the way for more nuanced understanding of human emotions.