EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Egocentric vision analysis is key for VR/AR and human-robot transfer.
• Existing methods focus on 'how' to interact, neglecting 'when' critical contact/separation occurs.
• Current temporal action localization (TAL) methods lack precision for fine-grained interaction moments and require category annotations.

Purpose of the Study:

• To address the underexplored problem of temporal interaction localization (TIL) in egocentric videos.
• To propose a novel zero-shot approach, EgoLoc, for accurately localizing hand-object contact and separation timestamps.
• To enable more immersive mixed reality experiences and improve robotic motion planning.

Main Methods:

• Developed EgoLoc, a zero-shot approach for temporal interaction localization (TIL).
• Introduced hand-dynamics-guided sampling for generating high-quality visual prompts.
• Leveraged vision-language models to identify contact/separation attributes and localize timestamps, incorporating closed-loop feedback for refinement.

Main Results:

• EgoLoc successfully localizes hand-object contact and separation timestamps in egocentric videos.
• The method achieves plausible temporal interaction localization (TIL) without requiring object masks or verb-noun taxonomies.
• Demonstrated generalizable zero-shot implementation across public datasets and novel benchmarks.

Conclusions:

• EgoLoc provides a robust solution for temporal interaction localization in egocentric vision.
• The approach effectively facilitates downstream applications in egocentric vision and robotic manipulation.
• EgoLoc's zero-shot capability enhances its applicability and generalizability in real-world scenarios.