Multi-Scale 2D Temporal Adjacency Networks for Moment Localization With Natural Language

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Untrimmed video moment retrieval by natural language is challenging due to complex temporal contexts.
• Existing methods often fail to adequately consider the relationships between different temporal moments within a video.

Purpose of the Study:

• To propose a novel framework for precise moment localization in untrimmed videos using natural language queries.
• To effectively model and leverage temporal contexts at multiple scales for improved retrieval accuracy.

Main Methods:

• Modeling temporal context using predefined 2D maps representing moment start times and durations across different temporal scales.
• Developing a Multi-Scale Temporal Adjacency Network (MS-2D-TAN), a single-shot framework for encoding multi-scale temporal adjacencies.
• Learning discriminative features for matching video moments with referring expressions.

Main Results:

• The proposed MS-2D-TAN framework demonstrates superior performance in video moment localization.
• Outperformed state-of-the-art methods on challenging benchmarks like Charades-STA, ActivityNet Captions, and TACoS.
• Effectively captures and utilizes adjacent temporal contexts at various scales.

Conclusions:

• The MS-2D-TAN effectively addresses the challenge of retrieving specific moments from untrimmed videos by incorporating multi-scale temporal context.
• The proposed 2D temporal map modeling and network architecture offer a significant advancement in natural language video moment retrieval.