A Human Intention and Motion Prediction Framework for Applications in Human-Centric Digital Twins

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Human-machine collaboration in manufacturing requires understanding human intention for seamless task execution.
• Current foundation models struggle with complex task anticipation and context-aware motion generation.

Purpose of the Study:

• To propose a modular framework for structuring task knowledge and engineering context-rich prompts.
• To guide Vision-Language Models (VLMs) in understanding and predicting human intention in semi-structured manufacturing environments.

Main Methods:

• Developed a modular framework for task knowledge structuring and context-rich prompt engineering.
• Implemented a Rolling Context Window strategy using frame history and predicted states.
• Evaluated performance across three use cases of varying complexity, incorporating egocentric video and historical data.

Main Results:

• A critical tradeoff between prediction accuracy and latency was identified.
• The Rolling Context Window strategy balanced performance and efficiency, outperforming single-image inputs and in-context learning.
• Egocentric video views increased performance by 10.7% for complex tasks.
• Historical pose, gaze data, and in-context examples enhanced short-term motion forecasting accuracy.

Conclusions:

• The proposed framework effectively enhances VLMs' ability to predict human intention in manufacturing.
• The Rolling Context Window strategy offers a practical solution for balancing accuracy and efficiency.
• Incorporating diverse data sources like egocentric video and historical sensor data is crucial for advanced human-robot collaboration.