LGCCT: A Light Gated and Crossed Complementation Transformer for Multimodal Speech Emotion Recognition

Area of Science:

• Multimodal machine learning
• Affective computing
• Natural Language Processing

Background:

• Speech emotion recognition (SER) is crucial, utilizing both acoustic and linguistic cues.
• Existing SER models often require extensive parameters and struggle with variable-length inputs.
• Efficiently fusing multimodal speech data for SER remains a challenge.

Purpose of the Study:

• To develop a lightweight and effective model for multimodal speech emotion recognition.
• To improve generalization to unseen data of varying lengths.
• To balance high performance with reduced computational overhead.

Main Methods:

• Proposed LGCCT (Light Gated and Crossed Complementation Transformer) for multimodal SER.
• Extracted acoustic features using CNN-BiLSTM and textual features using BiLSTM.
• Implemented a gate-control mechanism for balanced modality fusion and length-scaled dot product for attention to enhance generalization.

Main Results:

• Achieved an 81.0% F1 score on the CMU-MOSEI dataset with only 0.432M parameters.
• Demonstrated significant improvement in performance-parameter balance compared to baselines.
• Ablation studies confirmed model effectiveness and scalability to variable sequence lengths, showing ~20% improvement without length-scaled dot product.

Conclusions:

• LGCCT offers an efficient and effective solution for multimodal speech emotion recognition.
• The proposed methods enhance generalization and reduce computational complexity.
• The model shows promise for real-world applications requiring robust emotion recognition from speech.