KE-VUM: Knowledge-enhanced video understanding model for fine-grained movie description

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Fine-grained movie description is hindered by limited annotations and cross-modal semantic gaps, particularly in complex narratives.
• Current video description models often fail to utilize structured script knowledge, leading to superficial semantics and character misidentification.

Purpose of the Study:

• To develop a knowledge-enhanced video understanding model (KE-VUM) for improved fine-grained movie description.
• To address challenges in semantic gaps and narrative complexity in video understanding.
• To enhance temporal and causal coherence in video descriptions by integrating script knowledge.

Main Methods:

• KE-VUM employs collaborative reasoning between a video-based multimodal backbone and a script knowledge base.
• A two-stage script-guided optimization refines draft descriptions: entity/relationship correction via a knowledge graph and narrative reconstruction aligned with plot progressions.
• Introduced the MovieClip benchmark for narrative-rich clips and VD-Eval, an LLM-based framework for evaluating Semantic Accuracy Score (SAS) and Narrative Coherence Score (NCS).

Main Results:

• KE-VUM demonstrated superior performance over strong baselines on the MovieClip benchmark.
• Achieved a 6.7% absolute improvement in BERTScore F1, indicating enhanced descriptive accuracy.
• Showcased robust performance in delayed-narrative scenarios, improving both semantic accuracy and narrative coherence.

Conclusions:

• KE-VUM effectively integrates script knowledge for more accurate and coherent fine-grained video descriptions.
• The proposed model overcomes limitations of existing methods by leveraging structured knowledge for deeper video understanding.
• The MovieClip benchmark and VD-Eval provide valuable resources for advancing research in narrative video understanding.