First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Forecasting user behavior and goals from first-person camera data is challenging.
• Existing trajectory forecasting methods often lack long-term semantic reasoning.

Purpose of the Study:

• To develop an algorithm, Darko, for incrementally modeling and forecasting long-term goals of a first-person camera wearer.
• To reason about semantic and future goal states far in space and time.

Main Methods:

• Utilizes an Online Inverse Reinforcement Learning (IRL) approach to learn from streaming first-person visual observations.
• Darko discovers user transitions, rewards, and goals from sequential data.

Main Results:

• Darko forecasts goals more accurately than competing methods in both noisy and ideal conditions.
• The algorithm demonstrates theoretically and empirically no-regret performance.

Conclusions:

• Darko advances the state-of-the-art in long-term goal forecasting from egocentric video.
• The online IRL approach enables robust goal discovery from continuous user behavior streams.