SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioning.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Existing fine-grained video captioning methods struggle to capture subtle video dynamics and detailed information.
• Direct Preference Optimization (DPO) has limitations in vision-language model (VLM) training for video captioning.

Purpose of the Study:

• To enhance VLM performance in fine-grained video captioning using preference learning.
• To address limitations of DPO and improve the quality and efficiency of video descriptions.

Main Methods:

• Proposed a pipeline for constructing preference pairs using VLM properties and large language model assistance.
• Introduced Synergistic Preference Optimization (SynPO), a novel method outperforming DPO and its variants.
• SynPO prevents negative preferences from dominating, preserves language capabilities, and eliminates the need for a reference model.

Main Results:

• The proposed data construction pipeline and SynPO significantly and consistently improved fine-grained video captioning performance.
• Evaluated on Aurora-Cap, LLaVA1.6-7B-Video, and InterVL2-8B models, demonstrating consistent gains.
• Achieved high-quality captions and improved training efficiency.

Conclusions:

• The developed preference learning approach and SynPO offer a superior method for fine-grained video captioning.
• The method balances data quality and cost effectively.
• SynPO enhances training efficiency and caption quality in VLMs for video understanding.