Generalizable Egocentric Task Verification via Cross-Modal Hybrid Hypergraph Matching.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Egocentric Task Verification (ETV) traditionally uses video-to-video comparisons, limiting deployment flexibility.
• Recent ETV methods use textual rules but face cross-modal and hierarchical misalignment challenges.
• Existing ETV research often overlooks synthetic-to-realistic generalization and high-order matching interactions.

Purpose of the Study:

• To propose Generalizable Egocentric Task Verification (GETV) addressing synthetic-to-real generalization and complex cross-modal matching.
• To introduce EgoCross, a novel cross-domain ETV benchmark dataset for evaluating synthetic-to-real transfer.
• To develop a new method, Cross-modal Hybrid Hypergraph Matching (CHHM), for robust ETV.

Main Methods:

• Constructed the EgoCross dataset featuring synthetic training and realistic testing data across three task types.
• Developed Cross-modal Hybrid Hypergraph Matching (CHHM) to model cross-modal logical matching as hypergraph learning.
• Enhanced CHHM with prototype-based graph representation alignment to bridge the synthetic-to-real domain gap.

Main Results:

• Achieved new state-of-the-art performance on existing ETV datasets (EgoTV, CSV-NL) and the new EgoCross dataset.
• Demonstrated superior performance in both intra-domain and cross-domain ETV challenges.
• Validated the effectiveness of CHHM and prototype-based alignment in mitigating domain gaps.

Conclusions:

• The proposed GETV framework and CHHM method significantly advance egocentric task verification.
• The EgoCross dataset provides a crucial resource for developing and evaluating generalizable ETV models.
• The approach effectively handles cross-modal heterogeneity, hierarchical misalignment, and synthetic-to-real generalization challenges.