Multimodal Emotion Recognition Using Modality-Wise Knowledge Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Multimodal emotion recognition (MER) models typically combine unimodal features.
• Existing MER models face optimization imbalance issues where some modality encoders are undertrained.
• This imbalance can hinder overall model performance and accuracy.

Purpose of the Study:

• To propose a novel MER approach addressing the optimization imbalance problem.
• To enhance the performance of unimodal encoders within a multimodal framework.
• To leverage pre-trained unimodal models for improved MER.

Main Methods:

• Developed a multimodal emotion recognition (MER) system utilizing modality-wise knowledge distillation.
• Adapted unimodal encoders by leveraging knowledge from pre-trained unimodal emotion recognition models.
• Implemented a training strategy to mitigate optimization imbalance among modality encoders.

Main Results:

• The proposed MER method demonstrated superior performance compared to previous approaches on CREMA-D and IEMOCAP datasets.
• The knowledge distillation technique effectively overcame the optimization imbalance phenomenon.
• The approach showed compatibility and effectiveness when combined with existing MER methods.

Conclusions:

• Modality-wise knowledge distillation is an effective strategy for improving multimodal emotion recognition.
• The proposed method addresses a key limitation in current MER model training.
• This technique offers a promising direction for developing more robust and accurate MER systems.