Sequence as a Whole: A Unified Framework for Video Action Localization With Long-Range Text Query.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Understanding video content necessitates both spatial and temporal localization.
• Current methods like 3D CNNs have limited temporal context, while sequential methods struggle with dense cross-modal interactions.
• A unified framework for video action localization is lacking, hindering field development.

Purpose of the Study:

• To propose a unified framework for handling entire videos sequentially.
• To enable long-range and dense visual-linguistic interactions in an end-to-end manner.
• To improve performance across referring video action localization sub-tasks.

Main Methods:

• A unified framework processing videos sequentially.
• A lightweight relevance filtering based transformer (Ref-Transformer) incorporating relevance filtering attention and temporally expanded MLP.
• Efficiently highlighting text-relevant spatial regions and temporal clips for propagation.

Main Results:

• The proposed framework achieves state-of-the-art performance on referring video segmentation.
• The framework demonstrates superior results in temporal sentence grounding.
• The approach sets a new benchmark for spatiotemporal video grounding.

Conclusions:

• The unified framework effectively addresses limitations of existing methods.
• The Ref-Transformer enables efficient and effective long-range visual-linguistic interactions.
• The proposed method significantly advances the field of referring video action localization.