Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

EED-CL: Extended EEG Deformer with Contrastive Learning for Robust Emotion Recognition.

Bioengineering (Basel, Switzerland)·2026
Same author

EEG-Based Local-Global Dimensional Emotion Recognition Using Electrode Clusters, EEG Deformer, and Temporal Convolutional Network.

Bioengineering (Basel, Switzerland)·2025
Same author

Emotion Recognition Using Hierarchical Spatiotemporal Electroencephalogram Information from Local to Global Brain Regions.

Bioengineering (Basel, Switzerland)·2023
Same author

Multi-Camera-Based Sorting System for Surface Defects of Apples.

Sensors (Basel, Switzerland)·2023
Same author

Coronary Vessel Segmentation by Coarse-to-Fine Strategy Using U-nets.

BioMed research international·2021
Same author

Alzheimer's diagnosis using deep learning in segmenting and classifying 3D brain MR images.

The International journal of neuroscience·2020

Related Experiment Video

Updated: Jun 9, 2025

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception
05:48

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

1.4K

Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning.

Ju-Hwan Lee1, Jin-Young Kim1, Hyoung-Gook Kim2

  • 1Department of Intelligent Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Buk-gu, Gwangju 61186, Republic of Korea.

Bioengineering (Basel, Switzerland)
|October 25, 2024
PubMed
Summary

This study introduces a new multimodal emotion recognition system combining audio-visual data with electroencephalography (EEG) signals. The novel framework effectively integrates diverse data streams for improved emotion classification accuracy.

Keywords:
contrastive learningcross-attention mechanismemotion recognitionmultimodal learning

More Related Videos

Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology
05:38

Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology

Published on: June 29, 2021

2.3K
Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content
10:41

Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content

Published on: May 26, 2018

6.8K

Related Experiment Videos

Last Updated: Jun 9, 2025

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception
05:48

Author Spotlight: Investigating the Impact of Emotional Prosodies on Voice Recognition and Perception

Published on: August 9, 2024

1.4K
Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology
05:38

Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology

Published on: June 29, 2021

2.3K
Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content
10:41

Using Electroencephalography Measurements and High-quality Video Recording for Analyzing Visual Perception of Media Content

Published on: May 26, 2018

6.8K

Area of Science:

  • Computer Science
  • Neuroscience
  • Affective Computing

Background:

  • Multimodal emotion recognition leverages diverse data sources like physiological signals, visual cues, and audio-visual content.
  • Existing methods face challenges in managing redundant or conflicting cross-modal information and capturing implicit correlations.
  • Effective integration of disparate data streams is crucial for accurate human emotion understanding.

Purpose of the Study:

  • To develop a novel multimodal emotion recognition framework integrating audio-visual features with electroencephalography (EEG) data.
  • To enhance emotion classification accuracy by effectively processing and fusing information from multiple modalities.
  • To address limitations in current methods regarding redundant/conflicting information and inter-modal correlations.

Main Methods:

  • Utilized modality-specific encoders for extracting spatiotemporal features from audio-visual and EEG data.
  • Employed contrastive learning to align features and capture inter-modal relationships.
  • Incorporated cross-modal attention mechanisms for effective feature fusion.

Main Results:

  • The proposed framework demonstrated high effectiveness in emotion recognition across multiple datasets.
  • Integration of audio-visual features with EEG data significantly improved classification accuracy.
  • The approach successfully captured inter-modal relationships and fused features effectively.

Conclusions:

  • The novel multimodal framework integrating audio-visual and EEG data offers a significant advancement in emotion recognition.
  • The method provides a robust solution for handling complex emotional states by leveraging complementary information from different modalities.
  • This research highlights the potential of combining neurophysiological data with behavioral cues for more accurate and comprehensive emotion understanding.