Multi-Modal Residual Perceptron Network for Audio-Video Emotion Recognition

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Machine Learning

Background:

• Emotion recognition is crucial for advancing human-computer interaction.
• Current deep neural network approaches often focus on multi-modal superiority, overlooking uni-modal advantages.
• Existing fusion strategies can be hindered by noisy, indirect information in fuzzy emotion recognition tasks.

Purpose of the Study:

• To address limitations in current multi-modal emotion recognition techniques.
• To develop a novel network architecture that effectively integrates information from multiple modalities.
• To improve the performance of audio-video emotion recognition, especially for ambiguous emotional states.

Main Methods:

• Proposed a multi-modal residual perceptron network for end-to-end learning.
• Developed a novel time augmentation technique for streaming digital movies.
• Utilized late fusion and end-to-end multi-modal network training strategies.

Main Results:

• Achieved a state-of-the-art average recognition rate of 91.4% on the Ryerson Audio-Visual Database of Emotional Speech and Song dataset.
• Reached an average recognition rate of 83.15% on the Crowd-Sourced Emotional Multi Modal Actors dataset.
• Demonstrated improved multi-modal feature representation and generalization capabilities.

Conclusions:

• The multi-modal residual perceptron network offers a generalized approach to multi-modal feature representation.
• This architecture shows potential for various multi-modal applications beyond audio-visual data.
• The findings suggest a new direction for handling noisy and fuzzy information in emotion recognition systems.