Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Procedural activities are goal-oriented sequences with ordering constraints.
• Task graphs represent these activities holistically but often rely on hand-crafted procedures.
• Existing methods for task graph extraction from videos are limited.

Purpose of the Study:

• To develop a gradient-based approach for directly estimating task graphs from data.
• To improve the accuracy and applicability of task graph learning in video understanding.
• To enhance procedural understanding and online mistake detection in egocentric videos.

Main Methods:

• Employed gradient-based maximum likelihood optimization to learn edge weights for task graphs.
• Developed a differentiable framework enabling feature-based prediction from embeddings.
• Validated the approach on CaptainCook4D, EgoPER, and EgoProceL datasets, and the Ego-Exo4D benchmark.

Main Results:

• Achieved significant improvements in F1 score (+14.5%, +10.2%, +13.6%) on three datasets.
• Demonstrated state-of-the-art performance on the Ego-Exo4D benchmark across five downstream tasks.
• Showcased substantial gains in online mistake detection (+19.8%, +6.4%) on challenging datasets.

Conclusions:

• The proposed differentiable task graph learning approach offers superior performance over prior methods.
• This framework advances video understanding and procedural analysis, particularly in egocentric settings.
• The method facilitates more robust and accurate identification of procedural steps and errors.