Electroencephalogram-Based Analysis of Monomodal and Multimodal Interaction in Mixed Reality Games

Mixed reality (MR) technologies enable users to experience computer-generated content within the physical environment through spatial computing and head-mounted displays. By supporting real-time interaction through speech, gesture, gaze, and movement, MR offers new opportunities for game design beyond productivity and educational applications. However, relatively few studies have examined interaction modalities in MR games. In this paper, we present the design and deployment of four MR games on the Microsoft HoloLens 2: three that use monomodal input (speech, gaze, or gesture) and one that uses multimodal input (speech, gaze, and gesture). We conducted a study with ten participants and evaluated player experience using subjective self-reports of task load, emotional engagement, and comfort alongside objective measures, namely brain activity data collected with a five-channel electroencephalogram (EEG) device. Our preliminary findings suggest two clusters of interaction modalities based on subjective measures, a pattern that is also reflected in the objective EEG measures. Our analysis combining subjective and EEG data indicates that interaction modality influences task load and emotional engagement. Additionally, our functional connectivity analysis showed links in activity across the prefrontal, temporal, and occipital brain regions for different input modalities in the MR games.