Related Experiment Video
Updated: Apr 5, 2026
Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury
Published on: May 15, 2016
MBDA: A modality-balanced framework with data augmentation and alignment for multimodal emotion recognition
Cheng Cheng1, Ruisi Shang1, Zixu Wang2
1Institute of Psychology and Brain Sciences, Liaoning Normal University, No. 850 Huanghe Road, Dalian, 116029, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Normal University, No. 850 Huanghe Road, Dalian, 116029, China.
This study introduces a Modality-Balanced framework with Data Augmentation and Alignment (MBDA) to improve multimodal emotion recognition (MER). MBDA enhances data diversity and cross-modal alignment, leading to more robust emotion state inference.
Area of Science:
- Artificial Intelligence
- Human-Computer Interaction
- Affective Computing
Background:
- Multimodal Emotion Recognition (MER) integrates diverse data sources to understand human emotions.
- Existing MER methods face challenges like modality imbalance, feature misalignment, and insufficient data diversity, limiting performance.
- These limitations hinder the robustness and generalization capabilities of current MER systems.
Purpose of the Study:
- To develop a novel framework, Modality-Balanced framework with Data Augmentation and Alignment (MBDA), to overcome limitations in MER.
- To enhance the accuracy and reliability of inferring human emotional states from multimodal data.
- To improve the robustness and generalization of MER models through advanced techniques.
Main Methods:
- Proposed MBDA framework integrating modality-aware augmentation, feature alignment, and counterfactual knowledge distillation.
- Employed progressive learning to unify these components.
- Utilized modality-aware augmentation to increase data diversity while maintaining semantic consistency.
Main Results:
- MBDA demonstrated superior performance on benchmark datasets (DEAP and SEED-IV).
- Achieved high accuracies: 93.86% (DEAP-A), 95.11% (DEAP-V), 91.02% (DEAP-AV), and 92.66% (SEED-IV).
- Consistently outperformed existing state-of-the-art MER methods.
Conclusions:
- The MBDA framework effectively addresses modality imbalance and cross-modal misalignment in MER.
- MBDA significantly improves the robustness and generalization of emotion recognition systems.
- The proposed approach represents a substantial advancement in the field of multimodal emotion recognition.
