Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human visual perception utilizes activity-silent and persistent activity mechanisms for efficient information processing.
• Existing video grounding methods often struggle with comprehensive semantic association between video content and cross-modal queries.
• Efficient video grounding is crucial for applications like video retrieval and content analysis.

Purpose of the Study:

• To design a novel network, UniSDNet, that effectively learns semantic associations for video grounding by mimicking human visual perception.
• To improve both static and dynamic modeling for enhanced video context comprehension and query relevance.
• To achieve state-of-the-art performance in Natural Language Video Grounding (NLVG) and Spoken Language Video Grounding (SLVG) while increasing inference speed.

Main Methods:

• Developed UniSDNet incorporating a novel residual structure (ResMLP) for enhanced static modeling and global interaction.
• Implemented dynamic modeling inspired by persistent activity mechanisms, using a video clip graph with 2D sparse temporal masking.
• Employed a multi-kernel Temporal Gaussian Filter and element-level filtering convolutions for sophisticated context clue expansion and processing.

Main Results:

• UniSDNet achieves state-of-the-art (SOTA) performance on multiple NLVG and SLVG datasets.
• New records set include 38.88% R@1, IoU@0.7 on ActivityNet Captions and 40.26% R@1, IoU@0.5 on TACoS.
• The model demonstrates 1.56x faster inference speed compared to strong multi-query benchmarks.

Conclusions:

• UniSDNet offers a unified approach for efficient video grounding, effectively integrating static and dynamic information.
• The network's design, inspired by human visual perception, significantly enhances semantic understanding and context comprehension.
• The introduction of new SLVG datasets and the model's efficiency contribute to advancing the field of video grounding.