Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration.

Area of Science:

• Computer Science
• Human-Computer Interaction
• Virtual Reality

Background:

• Visuo-haptic mixed reality (MR) integrates virtual objects into real scenes for sight and touch interaction.
• Commodity haptic devices often cause visual obstruction and misalignment, hindering realistic integration of virtual tools and user hands.
• Existing MR systems struggle with seamless hand and virtual tool integration due to haptic device limitations.

Purpose of the Study:

• To propose a novel MR paradigm enabling users to interact with virtual objects through sight and touch using commodity haptic devices.
• To address and resolve visual obstruction and misalignment issues caused by haptic devices in MR environments.
• To achieve visually consistent integration of the user's hand and virtual tools within the mixed reality scene.

Main Methods:

• Developed a solution involving four key technical steps: hand color-based segmentation, haptic device tracking-based segmentation, background repainting using image-based models, and misalignment-free hand compositing.
• Implemented a proof-of-concept system demonstrating the proposed MR paradigm.
• Evaluated the impact of obstruction and misalignment correction on user performance.

Main Results:

• Successfully created a functional proof-of-concept visuo-haptic MR system.
• Demonstrated the ability for users to touch and interact with virtual objects within a real-world context.
• Quantified the performance improvements resulting from the implemented obstruction and misalignment correction techniques.

Conclusions:

• The proposed novel MR paradigm effectively overcomes visual obstruction and misalignment challenges posed by commodity haptic devices.
• The system allows for visually consistent and intuitive interaction with virtual objects in mixed reality.
• The developed techniques significantly enhance user performance in visuo-haptic mixed reality applications.