An Early Warning System Based on Visual Feedback for Light-Based Hand Tracking Failures in VR Head-Mounted Displays

Area of Science:

• Human-Computer Interaction
• Virtual Reality Systems
• Computer Vision

Background:

• Virtual Reality (VR) Head-Mounted Displays (HMDs) utilize built-in cameras for hand tracking, but accuracy is limited by hardware and software, impacting user experience.
• Hand skeleton detection algorithms in VR systems can fail due to inherent limitations, leading to interaction disruptions and user frustration.

Purpose of the Study:

• To develop and evaluate a visual feedback mechanism for an early warning system detecting hand skeleton recognition failures in VR HMDs.
• To enhance the usability of VR systems by proactively alerting users to impending hand tracking issues.

Main Methods:

• Two user studies were conducted involving virtual cup stacking and ball sorting tasks.
• The system monitored the VR HMD's hand tracking confidence and provided visual feedback when confidence levels dropped.
• Participants were warned visually before hand tracking algorithm failures occurred.

Main Results:

• The early warning system significantly improved the usability of the VR HMD system.
• User frustration was notably reduced when participants received advance warnings of tracking failures.
• The effectiveness of the visual feedback mechanism was demonstrated in distinct virtual task environments.

Conclusions:

• Proactive visual feedback for low hand tracking confidence enhances VR system usability and user satisfaction.
• This early warning system approach can be beneficial for any application relying on hand tracking, including robotics.
• Further research can explore adaptive feedback mechanisms tailored to specific user needs and task complexities.