Transformer-based emotion recognition in interactive art: A multimodal neural approach.

Area of Science:

• Affective neuroscience
• Human-computer interaction
• Digital aesthetics

Background:

• Understanding emotional responses to interactive digital art is crucial for bridging affective neuroscience and HCI.
• Previous research often used isolated EEG or self-report measures, limiting insights into temporal affective dynamics.
• This study integrates neural oscillatory features from EEG with subjective affect scores for a comprehensive analysis.

Purpose of the Study:

• To develop and validate a multimodal approach for modeling affective changes induced by interactive digital art.
• To investigate the relationship between electroencephalography (EEG) band-specific oscillations and subjective emotional states.
• To assess the efficacy of a Transformer-based deep learning model in predicting affective shifts.

Main Methods:

• Utilized a publicly available dataset with pre- and post-interaction EEG recordings and Positive and Negative Affect Scale (PANAS) scores.
• Pre-processed EEG data using independent component analysis, bandpass filtering, and z-score normalization.
• Trained a multi-output Transformer regression model to predict changes in positive and negative affect from EEG band-wise change features.

Main Results:

• The Transformer model outperformed LSTM and Random Forest baselines, achieving R² = 0.162.
• Delta-band oscillations were strongly associated with affective recovery.
• Changes in beta and gamma activity correlated with increased positive affect, while negative affect significantly decreased post-interaction.

Conclusions:

• EEG band-change features are effective for modeling affective variations in interactive art contexts.
• The multimodal approach provides methodological insights for affective computing in digital environments.
• Findings integrate emotion regulation theory, affective aesthetics, and deep learning for enhanced understanding of user experience.