Multi-Task Learning of Object States and State-Modifying Actions From Web Videos

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Understanding object state changes and actions is crucial for AI.
• Existing methods struggle with uncurated, long-form web videos.

Purpose of the Study:

• To develop a self-supervised model for temporal localization of object state changes and actions.
• To create an efficient multi-task architecture for joint learning.
• To introduce a large-scale dataset for evaluating these capabilities.

Main Methods:

• A self-supervised model leveraging causal ordering (initial state → action → end state).
• Exploration and identification of effective multi-task network architectures.
• Collection of the ChangeIt dataset (2600+ hours, 34K state changes).

Main Results:

• The multi-task model achieved a 40% relative improvement over prior methods.
• Significant outperformance compared to image-based and video-based zero-shot models.
• Effective learning on both COIN and the new ChangeIt dataset.

Conclusions:

• Self-supervised learning with causal signals is effective for action and state change localization.
• Multi-task architectures enhance efficiency and performance.
• The ChangeIt dataset provides a valuable resource for future research.