MRTouch: Adding Touch Input to Head-Mounted Mixed Reality.

Area of Science:

• Human-Computer Interaction
• Mixed Reality Systems
• Input Technologies

Background:

• Current mixed reality (MR) systems often rely on indirect input methods.
• Integrating intuitive and precise touch interaction into MR environments remains a challenge.
• Existing modalities like voice and gestures can be complemented by tactile feedback.

Purpose of the Study:

• To introduce MRTouch, a novel multitouch input solution for head-mounted MR systems.
• To enable users to directly manipulate virtual interfaces on environmental surfaces as if they were touchscreens.
• To evaluate the accuracy and robustness of the proposed touch input technique.

Main Methods:

• Developed a system combining depth and infrared camera streams for finger tracking.
• Implemented real-time detection and tracking of surface planes.
• Integrated the solution on a commercial Microsoft HoloLens without additional hardware or calibration.
• Conducted performance evaluations with 17 participants across various surface orientations and materials.

Main Results:

• Achieved high input accuracy with an average positional error of 5.4 mm.
• Demonstrated a 95% button size accuracy of 16 mm.
• Showcased robust finger-tracking capabilities even during hand and head motion.
• Validated the system's performance across different surfaces and orientations.

Conclusions:

• MRTouch provides a robust and accurate multitouch input solution for head-mounted MR systems.
• The system seamlessly integrates touch interaction with existing MR modalities.
• MRTouch has the potential to enable intuitive on-world touch interactions in various MR applications.