Understanding Changes of Emotion and Cognitive Effort in VR

Understanding how cognitive and emotional states evolve during everyday activities is important for designing effective virtual reality (VR) systems for training and rehabilitation, such as for recovering from traumatic brain injury (TBI). However, less is known about how these internal states fluctuate during naturalistic VR tasks. In this work, we investigate cognitive and emotional state dynamics during a VR shopping activity designed for TBI rehabilitation. Twenty healthy participants performed budgeting, item selection, and decision-making tasks while interacting with a virtual coach in either a low-stimulus (quiet) or high-stimulus (busy) environment. During the task, participants reported on momentary mental workload while physiological signals were continuously recorded. After the session, participants retrospectively annotated perceived cognitive and emotional state changes over time. Our results showed higher perceived workload and physiological arousal in the busy environment, with galvanic skin response activity positively correlated with reported cognitive workload. We describe the implications of the research, its limitations, and directions for future work.