Beyond the Game: Multimodal Emotion Recognition Before, During, and After Gameplay

Area of Science:

• Human-Computer Interaction (HCI)
• Affective Computing
• Machine Learning for Emotion Recognition

Background:

• Understanding user emotional responses during interactive experiences, particularly serious games (SG), is crucial for advancing Human-Computer Interaction (HCI).
• Multimodal signals offer a rich source of data for accurate emotion recognition (ER).
• Personalization in AI systems requires robust methods for detecting user affective states.

Purpose of the Study:

• To explore and enhance emotion recognition (ER) by analyzing multimodal data during serious game (SG) engagement.
• To investigate the interplay between physiological signals, game logs (GL), affect dynamics, and personality traits (PT) for personalized HCI.
• To identify distinct patterns in multimodal data that contribute to accurate ER in interactive scenarios.

Main Methods:

• Utilized the BIRAFFE-2 dataset, comprising data from 76 participants during dynamic gameplay and audiovisual stimulations.
• Extracted features from electrocardiogram (ECG), electrodermal activity (EDA), accelerometer, gyroscope, game logs (GL), affect dynamics, and personality traits (PT).
• Applied various machine learning models, including Support Vector Machines (SVM), for emotion recognition.

Main Results:

• Achieved state-of-the-art performance in emotion recognition across different experimental scenarios, with an accuracy of 0.967 for Negative Affect in Optimal Game using SVM.
• Highlighted the significance of emotional states as indicators for personalized HCI.
• Revealed distinct data patterns, emphasizing the role of ECG-Derived Respiration features and the impact of past affectivity on current emotional states.

Conclusions:

• The study demonstrates the effectiveness of multimodal signal analysis for accurate emotion recognition in serious games.
• Findings underscore the potential for developing adaptive, affect-sensitive SG by integrating physiological data, game logs, and user traits.
• Proposed objective digital biomarkers derived from multimodal signals can enhance clinical understanding of emotional behavior during SG-based interventions.