DeTAL: Open-Vocabulary Temporal Action Localization With Decoupled Networks.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Pre-trained visual-language (ViL) models show promise in video understanding but struggle with open-vocabulary temporal action localization (OV-TAL) due to robustness issues.
• Adaptation methods like fine-tuning can cause misalignment between visual features and text descriptions for unseen actions, creating a detection-classification trade-off.

Purpose of the Study:

• To address the limitations of current ViL models in OV-TAL.
• To propose a robust and effective method that decouples action detection from classification for improved OV-TAL performance.
• To enable OV-TAL even without action category annotations during training.

Main Methods:

• Introduced DeTAL, a two-stage approach for OV-TAL.
• Decoupled action detection and action classification to avoid performance compromises.
• Adapted state-of-the-art close-set action localization methods for the OV-TAL task.
• Developed a new cross-dataset evaluation setting to assess zero-shot capabilities.

Main Results:

• DeTAL significantly improves performance in OV-TAL by avoiding the detection-classification trade-off.
• The method demonstrates effectiveness even when action category annotations are unavailable during training.
• Experimental results show DeTAL outperforms existing state-of-the-art methods on THUMOS14 and ActivityNet1.3 datasets.

Conclusions:

• DeTAL offers a simple yet effective solution for OV-TAL, enhancing robustness and performance.
• The proposed decoupling strategy successfully tackles the challenges of open-vocabulary action localization.
• DeTAL provides a strong baseline for future research in zero-shot video understanding and action localization.