Transformer-based emotion recognition in interactive art: A multimodal neural approach

Understanding how interactive digital art affects emotional states is essential to advance research into the interface between affective neuroscience and human-computer interaction. Although previous studies have used either EEG or self-reported measures to evaluate emotional responses, only a few have integrated both to investigate the temporal dynamics of affective change. This study addresses this gap by adopting a multimodal approach combining neural oscillatory features with subjective affective assessment. We analysed a publicly available dataset containing pre- and post-interaction EEG recordings across five canonical frequency bands (Delta-Gamma) together with affect scores derived from the Positive and Negative Affect Scale (PANAS). EEG signals were pre-processed using independent component analysis, bandpass filtering, and z-score normalisation. A multi-output Transformer-based regression model was trained to predict affective shifts (ΔPositive, ΔNegative) from EEG band-wise change features. Statistical analyses included paired t-tests, ordinary least squares and Lasso regression, and permutation-based feature importance estimation. The Transformer outperformed LSTM and Random Forest baselines, achieving an R² of 0.162 with an MSE of 36.2 and MAE of 5.42. Delta-band oscillations showed the strongest association with affective recovery, while changes in beta and gamma activity were significantly associated with increases in positive affect (p < 0.01). Negative affect decreased significantly following interaction (p = 0.0014, d = 0.867). The dual-output structure of the model enabled the simultaneous modelling of positive and negative affective change. Overall, these findings demonstrate the utility of EEG band-change features for modelling affective variation in interactive art settings. The study integrates perspectives from emotion regulation theory, affective aesthetics, and deep learning, and provides methodological implications for multimodal affective modelling in interactive digital environments.