A Gesture- and Voice-controlled Virtual Reality System for Immersive Home Design: System Design and User Experience...

Gesture- and voice-based interaction in VR has shown promise in various applications; however, existing systems are hindered by their emphasis on navigation tasks, dependence on two-handed gestures, lack of fine-grained precision, and testing on small, homogeneous populations. These limitations limit applicability to intricate, design-centric workflows. To address these limitations, we introduce a Gesture- and Voice-Controlled Virtual Reality System for immersive home design, which incorporates the multimodal fusion of hand-tracked gesture input and natural language instructions for precise furniture placement, material selection, and spatial adjustment. Three aspects of novelty in the system are (i) adaptive modes of interaction to support one-handed usage, numeric voice commands, and coarse-to-fine adjustment of precision; (ii) multimodal fusion architecture merging gesture, speech, and context information for sub-centimeter precision; and (iii) task-aware user experience evaluation framework tailored to home design processes. The development process follows a systematic pipeline, from requirements analysis and design of gesture and voice lexicons, to multimodal intent recognition, VR scene assembly with snapping and alignment, and iterative testing. Experimental results on a wide variety of participant populations demonstrate a 30% decrease in placement error and substantial improvements in usability and workload ratings over controller-based interaction. The results suggest the potential of multimodal VR for creating accessible and engaging home design experiences.