Multimodal Knowledge Distillation for Emotion Recognition.

Area of Science:

• Affective Computing
• Human-Computer Interaction
• Biomedical Signal Processing

Background:

• Multimodal emotion recognition leverages complementary physiological signals like EEG and EOG for improved accuracy.
• EEG-based methods face practical limitations due to cost and complexity, hindering real-world applications.
• Electrooculography (EOG) offers a more feasible alternative for practical emotion recognition.

Purpose of the Study:

• To develop a novel multimodal knowledge distillation framework for practical emotion decoding.
• To transfer knowledge from a combined EEG-EOG model to a simplified EOG-only model.
• To maintain high accuracy in emotion recognition while enhancing real-world applicability.

Main Methods:

• A multimodal fusion module was designed to extract and integrate heterogeneous features from EEG and EOG signals.
• A unimodal EOG student model was structurally aligned with a multimodal teacher model for effective knowledge distillation.
• A dynamic feedback mechanism was incorporated to optimize knowledge transfer based on performance metrics.

Main Results:

• The proposed framework achieved state-of-the-art classification performance on two datasets (DEAP and BJTU-Emotion).
• Accuracies for valence and arousal on the DEAP dataset were 70.38% and 60.41%, respectively.
• Accuracies for valence and arousal on the BJTU-Emotion dataset were 61.31% and 60.31%, respectively, with statistically significant improvements (p < 0.05).

Conclusions:

• The framework successfully transfers knowledge from multimodal (EEG-EOG) models to unimodal EOG models.
• The proposed method enhances the practicality of emotion recognition by enabling accurate EOG-only decoding.
• This approach expands the real-world applicability of emotion recognition technologies.