AudioGest: Gesture-Based Interaction for Virtual Reality Using Audio Devices.

Area of Science:

• Human-Computer Interaction
• Virtual Reality Systems
• Acoustic Sensing

Background:

• Current virtual reality (VR) systems rely on cameras, handles, and touchscreens for gesture interaction.
• Existing methods are limited by device form factors and cannot extend interaction to surfaces like walls or tables.
• A need exists for scalable and versatile interaction methods in VR.

Purpose of the Study:

• To introduce AudioGest, a novel system for extending VR interaction to arbitrary surfaces.
• To develop a portable, plug-and-play solution for gesture detection using audio signals.
• To demonstrate the potential of AudioGest in diverse VR applications.

Main Methods:

• AudioGest utilizes a set of microphone devices to detect audio signals from finger interactions (tapping, sliding) on surfaces.
• An audio synthesis-recognition pipeline, based on micro-contact dynamics simulation, generates modal audio for different materials.
• Synthetic audio mixed with real audio is used as training data to verify accuracy and effectiveness.

Main Results:

• The system successfully generates synthetic audio reflecting material properties and physical interactions.
• The proposed audio synthesis-recognition pipeline demonstrates accuracy and effectiveness for gesture detection.
• Desktop office applications were developed, showcasing AudioGest's scalability and versatility in VR.

Conclusions:

• AudioGest offers a portable and calibration-free solution for expanding VR interaction areas.
• The system leverages acoustic sensing to enable intuitive gesture input on surfaces like walls and tables.
• AudioGest presents a promising approach for enhancing the immersive and interactive capabilities of virtual reality.