Object-Agnostic Transformers for Video Referring Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Video referring segmentation aims to identify objects in videos using text descriptions.
• Prior methods often use separate modules for context modeling and alignment, limiting interaction and flexibility.
• These methods require task-specific expertise, hindering generalizability.

Purpose of the Study:

• To propose a novel Object-Agnostic Transformer-based Network (OATNet) for video referring segmentation.
• To enable simultaneous intra-modal and inter-modal learning without object detection or category-specific labeling.
• To improve flexibility and generality in video referring segmentation.

Main Methods:

• OATNet utilizes a multi-modal encoder to process sequences of textual and visual tokens directly.
• It simultaneously explores context and alignment within the encoder.
• A cascade segmentation network refines segmentation from coarse to fine-grained.

Main Results:

• OATNet outperforms existing state-of-the-art methods on A2D Sentences and J-HMDB Sentences datasets.
• The approach demonstrates effective joint learning of context and alignment.
• It achieves superior performance without relying on object detection.

Conclusions:

• OATNet offers a more flexible and generalizable approach to video referring segmentation.
• Simultaneous intra-modal and inter-modal learning is crucial for improved performance.
• The proposed method advances the field by directly processing pixel data and textual descriptions.