ActPrompt: In-Domain Feature Adaptation via Action Cues for Video Temporal Grounding.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Video temporal grounding aims to locate specific clips within videos, utilizing pre-trained vision-language models (VLMs).
• Directly applying VLMs trained on images to temporal grounding tasks creates a domain gap, leading to performance degradation.
• This domain shift hinders VLMs' ability to differentiate action-specific patterns from static objects.

Purpose of the Study:

• To address the challenges of domain adaptation and integrating action-sensitive information into VLMs for temporal grounding.
• To propose an efficient feature adaptation paradigm that mitigates high adaptation costs.
• To enhance the performance of temporal grounding tasks by improving VLM feature representation.

Main Methods:

• Introduced a preliminary in-domain fine-tuning paradigm for efficient feature adaptation before downstream training.
• Developed Action-Cue-Injected Temporal Prompt Learning (ActPrompt) to inject action cues into VLM image encoders.
• Implemented context-aware temporal prompt learning to leverage both action cues and temporal context.

Main Results:

• The proposed preliminary fine-tuning paradigm significantly improves performance through well-designed pretext tasks.
• ActPrompt effectively injects action cues, enabling VLMs to better identify action-sensitive visual patterns.
• Extensive experiments show ActPrompt enhances various state-of-the-art methods for temporal grounding tasks.

Conclusions:

• ActPrompt is an effective off-the-shelf training framework for video temporal grounding.
• The method successfully bridges the domain gap and improves the recognition of action-related patterns.
• ActPrompt offers notable improvements for diverse state-of-the-art temporal grounding approaches.