UMEDNet: a multimodal approach for emotion detection in the Urdu language

Area of Science:

• Affective computing
• Human-computer interaction
• Natural Language Processing

Background:

• Emotion detection is vital for human-computer interaction, affective computing, and health screening.
• Multimodal approaches integrating video, speech, and text enhance emotion recognition.
• Low-resource languages like Urdu lack systematic models for emotion detection.

Purpose of the Study:

• To propose the Urdu Multimodal Emotion Detection Network (UMEDNet) for Urdu emotion detection.
• To introduce the Urdu Multimodal Emotion Detection (UMED) corpus, a novel resource for multimodal emotion analysis in Urdu.
• To address the gap in emotion detection research for low-resource languages.

Main Methods:

• Developed UMEDNet, a multimodal network utilizing video, speech, and text inputs.
• Created the UMED corpus: a 17-hour annotated dataset of five basic emotions in Urdu.
• Employed state-of-the-art feature extraction: MTCNN/FaceNet (video), Wav2Vec2 (speech), XLM-Roberta (text).

Main Results:

• UMEDNet achieved an overall accuracy of 85.27% on the UMED corpus.
• Precision, recall, and F1 scores were consistently high and approximately equivalent across metrics.
• Feature fusion in common latent spaces effectively enhanced multimodal emotion prediction accuracy.

Conclusions:

• UMEDNet demonstrates strong performance in Urdu emotion detection, validating the multimodal approach.
• The UMED corpus is a foundational resource for future research in low-resource multimodal emotion analysis.
• Integrating multimodal data significantly improves emotion detection accuracy compared to unimodal methods.