Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Temporal grounding enables locating video segments using natural language queries.
• Current datasets lack evaluation for compositional generalization in temporal grounding.
• Compositionality in language allows novel semantics from known words.

Purpose of the Study:

• To systematically benchmark compositional generalizability in temporal grounding.
• To introduce a new Compositional Temporal Grounding task and datasets (Charades-CG, ActivityNet-CG).
• To propose a novel framework addressing limitations in current temporal grounding models.

Main Methods:

• Introduced a new Compositional Temporal Grounding task and dataset splits.
• Developed a variational cross-graph reasoning framework.
• Proposed an adaptive structured semantics learning approach for graph representations.
• Introduced a challenging setting with unseen components in novel compositions.

Main Results:

• Existing models fail to generalize to novel word combinations in temporal grounding.
• The proposed framework demonstrates superior compositional generalizability.
• The approach effectively handles queries with novel combinations of seen words and unseen words.

Conclusions:

• Compositional structure in videos and language is crucial for generalization.
• The variational cross-graph reasoning framework enhances fine-grained semantic correspondence.
• The proposed methods advance the field of temporal grounding and compositional understanding.