Transferring Knowledge From Text to Video: Zero-Shot Anticipation for Procedural Actions

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Robots struggle to generalize activity recognition to novel tasks.
• Learning from video data alone is often insufficient for robust generalization.

Purpose of the Study:

• To develop a hierarchical model for robots to recognize and predict unseen activities from video using text-based knowledge.
• To enable robots to generalize instructional knowledge from large text corpora to video understanding tasks.

Main Methods:

• A hierarchical model was developed to learn and transfer instructional knowledge from text to video.
• The model processes video segments to recognize and predict future actions in natural language.
• Introduced the Tasty Videos Dataset V2 (4022 recipes) for zero-shot learning, recognition, and anticipation tasks.

Main Results:

• The model successfully recognizes and predicts plausible actions multiple steps into the future.
• Demonstrated strong generalization capabilities, even with limited video training data.
• Achieved robust performance across various evaluation metrics for video understanding.

Conclusions:

• Learning from text significantly enhances a robot's ability to generalize to new activities.
• The proposed hierarchical approach offers a promising direction for robot learning and video prediction.
• The Tasty Videos Dataset V2 facilitates research in zero-shot learning for instructional videos.