Generative-augmented multimodal neuroaffective decoding for product evaluation: toward neurofeedback-driven human-in-the-loop co-design
1Department of Industrial Design, School of Mechanical Engineering and Automation, Northeastern University, Shenyang, China.
Background:
Kansei engineering traditionally relies on self-reported evaluations of product impressions, which are susceptible to contextual influences, recall bias, and assessment timing. Multimodal psychophysiological measurements offer complementary insights into affective responses, but integrating heterogeneous signals with limited sample sizes remains challenging. This study proposes Generative-Augmented Multimodal Affective Fusion (GAMAF) to predict perceived attractiveness, innovativeness, and friendliness in product evaluation.
Methods:
GAMAF integrates feature-space conditional Wasserstein generative adversarial learning with gradient penalty (cWGAN-GP), cross-modal attention, graph-based interactions, and adaptive modality weighting. Fifty participants evaluated three automobile cockpit designs, yielding 150 participant-stimulus trials after multimodal quality control. Electroencephalography (EEG), eye tracking (ET), and peripheral physiological signals were integrated and evaluated using nested participant-wise cross-validation.
Results:
GAMAF achieved mean outer-fold R² values of 0.810, 0.850, and 0.840 for attractiveness, innovativeness, and friendliness, respectively, each with a standard deviation of 0.016. These values exceeded those of the Multimodal Transformer baseline (0.75, 0.78, and 0.76, respectively). Across the three targets, GAMAF reduced aggregate mean absolute error (MAE) and root mean squared error (RMSE) by approximately 29.6% and 29.7%, respectively. Ablation analyses supported the integrated architecture, while adaptive modality weights revealed target-dependent patterns.
Discussion:
GAMAF demonstrates the potential of generative-augmented multimodal fusion for decoding Kansei-related product impressions. However, performance comparisons remain descriptive because baseline variability and participant-level uncertainty were not quantified. Further validation on unseen product designs and closed-loop design optimization is needed to establish its generalizability and practical utility.