Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Hand trajectory prediction (HTP) is crucial for AR, robotics, and assistive tech.
• Existing HTP methods face challenges like limited targets, modality gaps, and entangled head-hand motion.

Purpose of the Study:

• To develop Uni-Hand, a universal framework for accurate and versatile hand motion forecasting.
• To address limitations of current HTP methods by incorporating multi-modal inputs and multi-target predictions.

Main Methods:

• Uni-Hand harmonizes modalities using vision-language fusion and task-aware text embeddings.
• A dual-branch diffusion model concurrently predicts head and hand movements.
• Target indicators enable prediction of specific joint waypoints (wrist, fingers) and hand-object interaction states.

Main Results:

• Uni-Hand achieves state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting.
• Experimental results on public datasets and new benchmarks validate its effectiveness.
• The framework demonstrates successful human-robot policy transfer and enhanced action recognition.

Conclusions:

• Uni-Hand offers a comprehensive solution for egocentric hand motion forecasting.
• Its multi-modal, multi-target approach enhances prediction accuracy and downstream task applicability.
• The framework shows significant potential for advancing robotics and human-computer interaction.