A comprehensive study on bilingual and multilingual speech emotion recognition using a two-pass classification...

Area of Science:

• Speech processing and machine learning for human-computer interaction.
• Multilingual emotion recognition and natural language processing.

Background:

• Existing speech emotion recognition (SER) studies predominantly focus on single languages.
• A gap exists in effectively recognizing emotions across multiple languages within a single system.
• This limitation hinders the development of truly universal human-computer interaction systems.

Purpose of the Study:

• To develop and evaluate an effective method for bilingual speech emotion recognition.
• To extend the proposed method for multilingual speech emotion recognition.
• To compare the performance of deep learning models (DNN, CNN) with i-vector features in a two-pass classification scheme.

Main Methods:

• A novel two-pass classification scheme: spoken language identification followed by language-specific speech emotion recognition.
• Utilized deep learning models: fully connected deep neural networks (DNN) and convolutional neural networks (CNN).
• Employed the i-vector paradigm for feature extraction on English (IEMOCAP) and German (FAU Aibo) corpora.

Main Results:

• The proposed two-pass method significantly outperformed a baseline bilingual approach.
• Achieved high unweighted average recalls (UARs): 64.0% (DNN) and 62.0% (CNN) on IEMOCAP; 61.14% (DNN) and 59.8% (CNN) on FAU Aibo.
• Extended to multilingual recognition (English, German, Japanese) with promising results, showing no significant UAR difference compared to monolingual classifiers.

Conclusions:

• The proposed two-pass method offers a powerful and effective solution for bilingual and multilingual speech emotion recognition.
• Deep learning and i-vectors are effective for cross-lingual emotion analysis.
• This approach advances the development of more sophisticated and globally applicable human-computer interaction technologies.