GestOnHMD: Enabling Gesture-based Interaction on Low-cost VR Head-Mounted Display

Area of Science:

• Human-Computer Interaction
• Virtual Reality Systems
• Acoustic Signal Processing

Background:

• Low-cost virtual reality (VR) head-mounted displays (HMDs) integrated with smartphones have increased VR accessibility.
• Current mobile VR systems often suffer from limited interactivity, hindering user experience.

Purpose of the Study:

• To introduce GestOnHMD, a novel gesture-based interaction technique for mobile VR.
• To develop a gesture-classification pipeline using smartphone microphones for detecting on-surface gestures on VR headsets.

Main Methods:

• Conducted a gesture-elicitation study on Google Cardboard to gather user-defined gestures.
• Selected gestures based on user preference and signal detectability for front, left, and right surfaces.
• Trained a deep-learning pipeline on acoustic data from 18 users performing selected gestures.

Main Results:

• Identified and selected 15, 9, and 9 distinct gestures for the front, left, and right surfaces, respectively.
• Developed a gesture-classification pipeline capable of detecting and recognizing on-surface gestures.
• Demonstrated GestOnHMD in real-time, collecting user-defined gesture-referent mappings.

Conclusions:

• GestOnHMD offers a viable solution for enhancing interactivity in mobile VR systems.
• Leveraging smartphone microphones for gesture recognition provides a low-cost and accessible interaction method.
• Further research into gesture-referent mappings can optimize GestOnHMD for diverse VR applications.