HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality

Area of Science:

• Human-Computer Interaction
• Computer Vision
• Machine Learning

Background:

• Hand and head movements are primary input methods in extended reality (XR).
• Existing XR models often focus on single modalities (hand or head) or specific applications, limiting generalizability.
• There is a need for methods that can jointly model complex hand and head movements in XR.

Purpose of the Study:

• To introduce HaHeAE, a novel self-supervised method for learning generalizable joint representations of hand and head movements in XR.
• To demonstrate the effectiveness of HaHeAE in improving reconstruction quality and enabling new XR applications.
• To showcase the potential of self-supervised learning for modeling human behavior in XR.

Main Methods:

• HaHeAE utilizes a self-supervised autoencoder (AE) architecture.
• It incorporates a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder.
• A diffusion-based decoder reconstructs the original hand-head movement signals.

Main Results:

• HaHeAE significantly outperforms existing self-supervised methods, achieving up to 74.1% improvement in reconstruction quality.
• The method demonstrates generalizability across diverse users, activities, and XR environments.
• HaHeAE enables novel applications like interpretable hand-head cluster identification and generative movement modeling.

Conclusions:

• HaHeAE is an effective self-supervised method for jointly learning hand and head movement representations in XR.
• The proposed approach offers superior performance and generalizability compared to prior methods.
• Self-supervised learning holds significant potential for advancing the understanding and application of human behavior in XR.