RefZVC: Refinable Zero-Shot Video Captioning by Test-Time Reinforcement Polishing

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Zero-shot image captioning (IC) using visual language models (VLMs) and large language models (LLMs) has advanced.
• Adapting these methods for zero-shot video captioning (VC) without paired supervision remains challenging.

Purpose of the Study:

• To develop a novel framework for zero-shot video captioning that addresses the lack of paired video-text data.
• To introduce a test-time reinforcement polishing paradigm for refining video captions.

Main Methods:

• Proposing Refinable Zero-shot VC (RefZVC) framework incorporating temporal dependency modeling.
• Designing an Adaptive Frame Skipping (AdaSkip) module to select keyframes.
• Implementing a Multi-granularity Reinforcement Polishing (MRP) mechanism with Gaussian Kernel Cache (GKC) for iterative caption refinement.

Main Results:

• RefZVC effectively captures long-term video context and refines captions via reward feedback.
• The MRP mechanism enables sentence-level and entity-level caption polishing.
• RefZVC demonstrates superior zero-shot generalization performance on MSVD, MSR-VTT, and VATEX benchmarks.

Conclusions:

• The proposed RefZVC framework with test-time reinforcement polishing significantly improves zero-shot video captioning.
• The AdaSkip and MRP modules are key innovations for handling temporal dynamics and context.
• This approach offers a promising direction for unsupervised video understanding tasks.