Hands-Free User Interface for AR/VR Devices Exploiting Wearer's Facial Gestures Using Unsupervised Deep Learning

Area of Science:

• Human-computer interaction
• Augmented reality (AR) technology
• Biomedical engineering

Background:

• Developing intuitive user interfaces (UIs) for augmented reality (AR) headset environments presents significant challenges.
• Existing AR interfaces often require manual input, hindering seamless user experience.

Purpose of the Study:

• To propose and validate a novel hands-free UI for AR headsets that utilizes facial gestures for user intention recognition.
• To develop a robust system for detecting and classifying facial gestures using custom sensor technology and deep learning.

Main Methods:

• A custom sensor detecting skin deformation via infrared diffusion characteristics was developed to capture facial gestures.
• A deep neural network classifier, comprising a spatiotemporal autoencoder and deep embedded clustering, was designed for unsupervised gesture recognition.
• The UI system was integrated into a commercial AR headset, and experiments were conducted on online sensor data.

Main Results:

• The proposed system achieved an average accuracy of 95.4% in recognizing user commands through facial gestures.
• The unsupervised learning approach effectively classified complex skin deformation patterns into distinct user intentions.
• Successful implementation and verification of the hands-free UI within a commercial AR headset were demonstrated.

Conclusions:

• Facial gesture recognition using infrared-based skin deformation sensing offers a viable and accurate method for hands-free control in AR headsets.
• The developed deep learning model provides an effective solution for real-time, unsupervised classification of user intentions from physiological signals.
• This research advances the field of AR by enabling more natural and immersive human-computer interaction within headset environments.