Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual...

Area of Science:

• Affective computing
• Human-computer interaction
• Educational technology

Background:

• Assessing user affective states is crucial for personalized virtual learning experiences.
• Traditional emotion recognition often relies on complete multimodal data, which is rare in real-world virtual learning environments.
• Existing methods struggle with the dynamic and incomplete nature of sensory data in these settings.

Purpose of the Study:

• To develop and evaluate methods for recognizing user affective states in virtual learning environments using continuous arousal and valence dimensions.
• To address the challenge of missing data across audio, text, and video modalities.
• To enable robust emotion recognition despite incomplete or variable data availability.

Main Methods:

• Utilized multimodal information (audio, text, video) for affective state recognition.
• Proposed feature-level and decision-level fusion approaches to handle missing data.
• Investigated techniques inspired by neural network dropout and recurrent neural network variable input lengths for modality recognition.

Main Results:

• Demonstrated effective recognition of continuous arousal and valence dimensions even with missing modalities.
• Showcased the adaptability of proposed fusion methods to varying data availability in virtual learning scenarios.
• Validated the innovative approach of representing emotions in a continuous space within educational contexts.

Conclusions:

• The proposed multimodal emotion recognition system effectively assesses user affective states in virtual learning environments.
• The methods developed can robustly handle missing data, making them practical for real-world applications.
• Representing emotions continuously and utilizing available modalities enhances the understanding of user experience in virtual education.