Gesture2Text: A Generalizable Decoder for Word-Gesture Keyboards in XR Through Trajectory Coarse Discretization and...

Area of Science:

• Human-Computer Interaction
• Virtual and Augmented Reality
• Machine Learning

Background:

• Word-gesture keyboards (WGK) are key interactions in Extended Reality (XR), but decoding word-gesture trajectories is complex due to diverse interaction modes and data patterns.
• Template-matching methods like SHARK2 are common but struggle with noisy trajectories, while conventional neural decoders require extensive data and expertise.
• Existing methods face challenges in accuracy, data requirements, and implementation complexity for WGK systems.

Purpose of the Study:

• To develop a novel, generalizable neural decoder for WGK systems that combines ease of implementation with high decoding accuracy.
• To address the limitations of existing template-matching and neural-network-based decoders for WGK trajectory decoding.
• To create a ready-to-use WGK decoder applicable across various XR environments and interaction types.

Main Methods:

• Proposed a generalizable neural decoder enabled by pre-training on large-scale, coarsely discretized word-gesture trajectories.
• Evaluated the decoder's performance across mid-air and on-surface WGK systems in augmented reality (AR) and virtual reality (VR).
• Quantized the pre-trained decoder to 4 MB and measured its real-time execution performance on a Quest 3 device.

Main Results:

• Achieved a robust average Top-4 accuracy of 90.4% across four diverse datasets, demonstrating broad generalizability.
• Significantly outperformed SHARK2 by 37.2% and a conventional neural decoder by 7.4%.
• The quantized decoder maintained accuracy while being only 4 MB and executing in 97 milliseconds on Quest 3, enabling real-time performance.

Conclusions:

• The proposed pre-trained neural decoder offers a highly accurate and efficient solution for WGK trajectory decoding in XR.
• This approach overcomes the limitations of existing methods, providing a generalizable and easy-to-implement decoder for diverse AR and VR applications.
• The decoder's small size and real-time capability make it suitable for deployment on resource-constrained XR hardware.