Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Apr 5, 2026

Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury
05:51

Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury

Published on: May 15, 2016

9.5K

MBDA: A modality-balanced framework with data augmentation and alignment for multimodal emotion recognition.

Cheng Cheng1, Ruisi Shang1, Zixu Wang2

  • 1Institute of Psychology and Brain Sciences, Liaoning Normal University, No. 850 Huanghe Road, Dalian, 116029, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Normal University, No. 850 Huanghe Road, Dalian, 116029, China.

Neural Networks : the Official Journal of the International Neural Network Society
|April 3, 2026
PubMed
Summary

Related Concept Videos

Labeling Emotion01:20

Labeling Emotion

944
Emotional labeling is a cognitive process that involves identifying and naming one's emotions, such as anger, fear, happiness, or sadness. It allows individuals to recognize and express their internal emotional states, a critical aspect of emotional regulation and communication. Labeling emotions requires more than mere recognition; it also involves drawing upon memory and contextual cues to understand the current situation and apply a corresponding emotional label. For instance, feeling...
944

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Association of total sedentary behaviour and television viewing with risk of overweight/obesity, type 2 diabetes and hypertension: A dose-response meta-analysis.

Diabetes, obesity & metabolism·2019
Same author

AAV-ie enables safe and efficient gene transfer to inner ear cells.

Nature communications·2019
Same author

The effect of the overexpression of miR-490 on the biological function of prostate cancer cell PC-3.

Journal of B.U.ON. : official journal of the Balkan Union of Oncology·2019
Same author

Treatment response and outcome of children with T-cell acute lymphoblastic leukemia expressing the gamma-delta T-cell receptor.

Oncoimmunology·2019
Same author

Hypertension is a modifiable risk factor for osteonecrosis in acute lymphoblastic leukemia.

Blood·2019
Same author

Patients' experiences of coping with multiple chronic conditions: A qualitative descriptive study.

Journal of clinical nursing·2019
Same journal

Exploiting audio-visual modalities in videos: Object detection via multi-stage bilateral coupling network.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Reliability-aware modality completion with cross-modal distillation for federated learning with missing modalities.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

IGFD-Net: Illumination-guided frequency decoupling for polarization image fusion.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Multiple-Strategies dung beetle optimizer and its applications in engineering optimization and bankruptcy prediction.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Aggregating global-scale pixel-wise forgery cues within a graph.

Neural networks : the official journal of the International Neural Network Society·2026
Same journal

Finite-Time intermittent control for secure synchronization of Neutral-Type stochastic delayed neural networks under aperiodic DoS attacks.

Neural networks : the official journal of the International Neural Network Society·2026
See all related articles
This summary is machine-generated.

This study introduces a Modality-Balanced framework with Data Augmentation and Alignment (MBDA) to improve multimodal emotion recognition (MER). MBDA enhances data diversity and cross-modal alignment, leading to more robust emotion state inference.

Area of Science:

  • Artificial Intelligence
  • Human-Computer Interaction
  • Affective Computing

Background:

  • Multimodal Emotion Recognition (MER) integrates diverse data sources to understand human emotions.
  • Existing MER methods face challenges like modality imbalance, feature misalignment, and insufficient data diversity, limiting performance.
  • These limitations hinder the robustness and generalization capabilities of current MER systems.

Purpose of the Study:

  • To develop a novel framework, Modality-Balanced framework with Data Augmentation and Alignment (MBDA), to overcome limitations in MER.
  • To enhance the accuracy and reliability of inferring human emotional states from multimodal data.
  • To improve the robustness and generalization of MER models through advanced techniques.

Main Methods:

Keywords:
Counterfactual knowledge distillationData alignmentData augmentationMultimodal emotion recognition (MER)

Related Experiment Videos

Last Updated: Apr 5, 2026

Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury
05:51

Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury

Published on: May 15, 2016

9.5K
  • Proposed MBDA framework integrating modality-aware augmentation, feature alignment, and counterfactual knowledge distillation.
  • Employed progressive learning to unify these components.
  • Utilized modality-aware augmentation to increase data diversity while maintaining semantic consistency.

    • Main Results:

    • MBDA demonstrated superior performance on benchmark datasets (DEAP and SEED-IV).
    • Achieved high accuracies: 93.86% (DEAP-A), 95.11% (DEAP-V), 91.02% (DEAP-AV), and 92.66% (SEED-IV).
    • Consistently outperformed existing state-of-the-art MER methods.

    Conclusions:

    • The MBDA framework effectively addresses modality imbalance and cross-modal misalignment in MER.
    • MBDA significantly improves the robustness and generalization of emotion recognition systems.
    • The proposed approach represents a substantial advancement in the field of multimodal emotion recognition.