TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Object detection has seen advancements with Transformers like DETR.
• Performance of Transformer-based models in Video Object Detection (VOD) remains underexplored.
• Current VOD methods often rely on complex, hand-designed components and post-processing.

Purpose of the Study:

• To present TransVOD, the first end-to-end VOD system utilizing spatial-temporal Transformer architectures.
• To streamline the VOD pipeline by removing the need for optical flow models and relation networks.
• To eliminate post-processing steps like Seq-NMS through Transformer-based query design.

Main Methods:

• Introduced a temporal Transformer to aggregate spatial object queries and frame features.
• Developed a Temporal Query Encoder (TQE) for fusing object queries.
• Designed a Temporal Deformable Transformer Decoder (TDTD) for current frame detection.
• Proposed TransVOD++, integrating object-level information via dynamic convolution.
• Developed TransVOD Lite for faster inference by processing entire video clips.

Main Results:

• TransVOD significantly improved upon the deformable DETR baseline by 3-4% mAP on ImageNet VID.
• TransVOD++ achieved a new state-of-the-art accuracy of 90.0% mAP on ImageNet VID.
• TransVOD Lite demonstrated a strong speed-accuracy trade-off, reaching 83.7% mAP at ~30 FPS on a V100 GPU.

Conclusions:

• TransVOD offers a simplified and effective end-to-end framework for video object detection.
• The proposed temporal Transformer architecture is crucial for enhancing VOD performance.
• TransVOD++ and TransVOD Lite provide advanced solutions for high-accuracy and efficient video object detection, respectively.