[Research on bimodal emotion recognition algorithm based on multi-branch bidirectional multi-scale time perception].

Area of Science:

• Multimodal machine learning
• Affective computing
• Signal processing

Context:

• Human emotion recognition relies on integrating information from multiple modalities, primarily voice and facial expressions.
• Effectively fusing these distinct data streams presents a significant challenge in developing accurate emotion recognition systems.
• Existing methods often struggle to capture the complex temporal and multi-scale features inherent in emotional expressions.

Purpose:

• To propose a novel multi-branch bidirectional multi-scale time perception model for emotion recognition.
• To develop a two-modal feature dynamic fusion algorithm for enhanced feature integration.
• To improve the accuracy and effectiveness of emotion recognition by combining voice and facial expression data.

Summary:

• A multi-branch bidirectional multi-scale time perception model detects speech Mel-frequency spectrum coefficients and uses causal convolution for temporal correlation and multi-scale feature fusion.
• A two-modal feature dynamic fusion algorithm leverages AlexNet and overlapping maximum pooling for richer fusion of audio-visual emotion features.
• The proposed dual-modal emotion recognition model achieves state-of-the-art accuracy of 97.67% and 90.14% on public datasets.

Impact:

• The developed model demonstrates superior performance in emotion recognition compared to conventional approaches.
• It effectively captures and integrates crucial emotion-related features from both auditory and visual cues.
• This research advances the field of affective computing by providing a more accurate and robust method for dual-modal emotion recognition.