End-to-End Temporal Action Detection With Transformer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Temporal Action Detection (TAD) is crucial for video understanding but traditionally relies on complex, multi-stage pipelines.
• Existing methods often involve hand-designed components, hindering end-to-end learning and flexibility.
• These limitations necessitate more efficient and integrated approaches for accurate action instance identification.

Purpose of the Study:

• To propose TadTR, a novel end-to-end Transformer-based method for Temporal Action Detection.
• To overcome the limitations of complex pipelines and hand-designed operations in previous TAD methods.
• To enhance Transformer's locality awareness for improved video action recognition.

Main Methods:

• Introduced TadTR, an end-to-end Transformer architecture utilizing learnable action queries.
• Developed a temporal deformable attention module for selective focus on key video snippets.
• Incorporated a segment refinement mechanism and an actionness regression head for precise boundary and confidence prediction.

Main Results:

• TadTR achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP) as a self-contained detector.
• When combined with an action classifier, TadTR reaches 36.75% mAP on ActivityNet-1.3.
• The proposed method demonstrates remarkable performance with significantly lower computational cost compared to prior works.

Conclusions:

• TadTR offers a simplified, end-to-end solution for Temporal Action Detection.
• The Transformer-based approach with enhanced locality awareness proves effective for video understanding.
• TadTR sets a new benchmark for efficiency and performance in temporal action detection tasks.