A Hybrid Deep Learning Emotion Classification System Using Multimodal Data.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Speech Emotion Recognition

Background:

• Emotion classification is crucial for AI and personalized services.
• Existing single-modality approaches face challenges with intonation, text structure, and physiological signal variations.
• Korean language presents unique NLP challenges like subject omission and spacing.

Purpose of the Study:

• To propose a hybrid multimodal deep learning system (HDECS) for enhanced emotion classification in Korean.
• To address limitations of single-modality sentiment analysis in spoken contexts.
• To improve emotion recognition accuracy using multimodal data.

Main Methods:

• Retraining LSTM and CNN models for multimodal Korean data until prediction agreement exceeds 0.75.
• Generating emotional sentences from predictions to augment script data.
• Utilizing BERT for final emotion prediction on the enhanced dataset.

Main Results:

• The HDECS case model demonstrated superior performance compared to the KLUE/roBERTa model.
• Achieved improvements of 0.5 in Categorical Cross-Entropy (CCE), 0.09 in accuracy, and 0.11 in F1 score.
• The model effectively integrates multimodal information for robust emotion classification.

Conclusions:

• The proposed HDECS offers a significant advancement in Korean emotion classification.
• The hybrid approach is adaptable for sentiment classification across various languages and regional speech characteristics.
• HDECS shows promise for real-world applications requiring nuanced emotion understanding.