RefZVC: Refinable Zero-Shot Video Captioning by Test-Time Reinforcement Polishing

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Zero-shot image captioning (IC) using visual language models (VLMs) and large language models (LLMs) has advanced significantly.
• Adapting these methods for zero-shot video captioning (VC) without paired supervision remains a challenge.
• Test-time strategies offer potential for performance gains by utilizing additional testing time.

Purpose of the Study:

• To develop a novel framework for zero-shot video captioning (VC) that addresses the lack of paired video-text supervision.
• To introduce a test-time reinforcement polishing paradigm for refining generated video captions.
• To enhance the model's ability to capture long-term temporal dependencies in videos for improved captioning.

Main Methods:

• Proposing Refinable Zero-shot VC (RefZVC), a framework incorporating temporal dependency modeling.
• Designing an Adaptive Frame Skipping (AdaSkip) module to select diverse keyframes and skip redundant ones.
• Introducing a Multi-granularity Reinforcement Polishing (MRP) mechanism with Gaussian Kernel Cache (GKC) for iterative caption refinement using reward feedback.

Main Results:

• RefZVC effectively captures long-term video context and refines captions through a reward-feedback loop.
• The MRP mechanism leverages GKC for temporal dynamics and reuses historical context.
• MRP provides sentence-level and entity-level rewards for test-time caption polishing.

Conclusions:

• RefZVC demonstrates superior zero-shot generalization capabilities in video captioning.
• The proposed test-time reinforcement polishing approach significantly enhances VC performance.
• RefZVC outperforms existing zero-shot VC methods on benchmarks like MSVD, MSR-VTT, and VATEX.