SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

Area of Science:

• Human-Computer Interaction
• Extended Reality (XR)
• Sensor Fusion

Background:

• Mid-air gestures in Extended Reality (XR) often result in user fatigue and imprecision.
• Current egocentric vision-based methods for surface interaction in XR face challenges with hand tracking and surface plane estimation.
• Surface-based interactions offer enhanced accuracy, speed, and comfort compared to mid-air gestures.

Purpose of the Study:

• To introduce SurfaceXR, a novel sensor fusion approach for robust and precise surface-based interactions in XR.
• To overcome the limitations of existing egocentric vision-based methods for surface input in XR.
• To enable flexible and accurate on-surface interactions using complementary sensor data.

Main Methods:

• Developed SurfaceXR, a sensor fusion system combining headset-based hand tracking with smartwatch Inertial Measurement Unit (IMU) micro-vibration data.
• Designed the system for flexible operation using hand tracking, IMU sensing, or both modalities combined.
• Ensured robustness without explicit surface calibration by leveraging complementary positional and motion data.

Main Results:

• Validated SurfaceXR's effectiveness through a user study with 21 participants, demonstrating significant improvements in surface touch tracking.
• Achieved high accuracy in 8-class hand-surface gesture recognition, outperforming single-modality approaches.
• Showcased enhanced performance over single-modality methods for both hand tracking and IMU sensing.

Conclusions:

• SurfaceXR enables precise and robust surface-based interactions in XR by fusing hand tracking and IMU data.
• The system offers a flexible and calibration-free solution for augmenting touch tracking and gesture recognition on everyday surfaces.
• Demonstrated the potential of SurfaceXR through interactive applications in Augmented Reality (AR) and Virtual Reality (VR).