A novel transformer autoencoder for multi-modal emotion recognition with incomplete data.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Affective Computing

Background:

• Multi-modal signals are crucial for comprehensive emotion recognition.
• Missing modalities in real-world data severely degrade performance in emotion recognition tasks.
• Existing methods struggle with incomplete multi-modal datasets for emotion recognition.

Purpose of the Study:

• To propose the first transformer-based architecture for filling modality-incomplete data in multi-modal emotion recognition (MER).
• To develop a unified model, the Transformer Autoencoder (TAE), capable of handling partially observed multi-modal signals.
• To enhance the robustness and accuracy of emotion recognition systems when faced with missing data.

Main Methods:

• A novel Transformer Autoencoder (TAE) model is proposed, featuring modality-specific hybrid transformer encoders, an inter-modality transformer encoder, and a convolutional decoder.
• The modality-specific encoder integrates convolutional and transformer components to capture local and global context within each modality.
• The inter-modality encoder aligns cross-modal correlations and models long-range dependencies, while a regularization term aids in leveraging incomplete data.

Main Results:

• The TAE model achieved high accuracies on complete data (up to 96.33% on DEAP and SEED-IV datasets).
• Remarkable performance was maintained on incomplete data, with accuracies reaching 93.25% on DEAP and 81.76% on SEED-IV.
• The model demonstrated a significant advantage (5.61% improvement with 70% missing data) over state-of-the-art approaches for incomplete multi-modal learning.

Conclusions:

• The proposed Transformer Autoencoder (TAE) effectively addresses the challenge of missing modalities in multi-modal emotion recognition.
• TAE demonstrates superior performance compared to existing methods, particularly in scenarios with significant data incompleteness.
• This work offers a promising direction for robust emotion recognition systems in real-world, data-scarce environments.