Multimodal Feature Fusion Method for Unbalanced Sample Data in Social Network Public Opinion

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Social media analysis increasingly requires understanding multi-modal data beyond text, including voice and facial expressions.
• Multi-modal emotion recognition, particularly from speech, is crucial for accurate public opinion analysis but faces challenges with data imbalance.

Purpose of the Study:

• To develop and evaluate a multi-modal feature fusion method for emotion recognition using speech and text data.
• To address the challenge of sample disequilibrium in multi-modal datasets for improved emotion analysis.

Main Methods:

• Exploration of speech emotion feature retrieval techniques.
• Analysis of processing methods for imbalanced sample data.
• Development of a multi-modal feature fusion approach combining text and speech for imbalanced datasets.

Main Results:

• The proposed method achieved good fine-grained emotion recognition results on the IEMOCAP and MELD datasets.
• Effective fusion of multi-modal data, including speech and text, was demonstrated.

Conclusions:

• The developed multi-modal emotion recognition method provides a foundation for advanced social public opinion analysis.
• Addressing data imbalance is key to successful multi-modal emotion recognition in real-world social media contexts.