Putting Vision and Touch into Conflict: Results From a Multimodal Mixed Reality Setup

Area of Science:

• Multisensory perception
• Human-computer interaction
• Virtual reality

Background:

• Previous research on sensory conflict focused on low-level object properties.
• The dominance of vision or touch in perception remains an active area of investigation.
• Understanding sensory integration is crucial for designing effective virtual and augmented reality experiences.

Purpose of the Study:

• To investigate the dominance of vision versus touch when presented with conflicting information about high-level object shape properties.
• To explore the influence of explicit instructions on sensory weighting in multimodal perception.
• To introduce and validate a novel mixed-reality setup for studying sensory conflict.

Main Methods:

• Developed a mixed-reality system integrating real-time hand-tracking with a head-mounted display (HMD).
• Participants performed haptic exploration of virtual objects with conflicting visual and tactile information.
• Varied object properties (global shape), experimental settings (unimodal, multimodal congruent/incongruent), and participant instructions.

Main Results:

• Touch perception demonstrated a clear dominance over vision in determining perceived object shape, irrespective of sensory congruence.
• Instructions to bias modality weighting had a minimal effect on the observed touch dominance.
• Hand-tracking data provided insights into object exploration behaviors, highlighting the setup's potential for detailed analysis.

Conclusions:

• Touch plays a dominant role in perceiving object shape, even when vision provides conflicting cues.
• The findings suggest robust tactile processing that is resistant to explicit attentional modulation.
• The developed mixed-reality system offers a valuable tool for future research into sensory integration and exploration behavior.