Computational Modeling of Affective User Experience Using Multimodal Physiological and Behavioral Signals

Area of Science:

• Computational neuroscience
• Affective computing
• Machine learning for biosignal processing

Background:

• Multimodal affective modeling integrates diverse physiological signals for robust emotion recognition.
• Existing methods often lack a unified framework for processing heterogeneous biosignals.
• Accurate emotion recognition is crucial for understanding user experience and developing adaptive systems.

Purpose of the Study:

• To propose a reproducible computational protocol for multimodal affective modeling using physiological signals.
• To enable offline emotion recognition by integrating multiple biosignals (EEG, ECG, GSR) within a deep learning framework.
• To validate the effectiveness of a multimodal fusion workflow for computational affective modeling.

Main Methods:

• A five-step protocol: data collection, preprocessing, feature alignment, multimodal fusion, and evaluation.
• Utilized Deep Canonical Correlation Analysis (DCCA) for aligning heterogeneous feature spaces across modalities.
• Employed a multimodal fusion network for classifying affective states and emotional dimensions (valence-arousal).

Main Results:

• The protocol achieved 92.1% accuracy for user experience-affective state prediction.
• Attained a 94.2% F1-score for valence-arousal classification, outperforming baseline models.
• Demonstrated consistent superiority of the proposed multimodal fusion workflow on standard performance metrics (accuracy, precision, recall, F1-score, AUC).

Conclusions:

• The developed computational framework provides an effective approach for multimodal affective user experience modeling.
• The proposed protocol offers a reproducible method for integrating and fusing physiological signals for emotion recognition.
• Findings confirm the efficacy of the multimodal fusion workflow in benchmarking physiological data for affective computing.