Improving Video Summarization by Exploring the Coherence Between Corresponding Captions.

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Processing

Background:

• Current video summarization methods often produce incoherent summaries due to a lack of holistic segment considerations.
• Optimizing coherence in video summaries is challenging from a purely visual perspective.

Purpose of the Study:

• To propose a novel Language-guided Segment Coherence-Aware Network (LS-CAN) for generating more coherent video summaries.
• To leverage text modality for measuring and optimizing segment coherence in video summarization.

Main Methods:

• Developed the LS-CAN, integrating language-guided coherence into key segment recognition.
• Introduced a multi-graph correlated neural network (MGCNN) module to measure text coherence based on subjects, attributes, and actions.
• Utilized large language models to augment text coherence annotations for improved model performance.

Main Results:

• The LS-CAN significantly enhances video summary coherence.
• Experimental results show state-of-the-art improvements on the BLiSS dataset, with notable gains in F1 scores, tau, and rho metrics.
• Each proposed module within LS-CAN demonstrated effectiveness in improving summarization quality.

Conclusions:

• Integrating text coherence is a viable and effective strategy for improving video summarization.
• The LS-CAN framework offers a promising direction for creating higher-quality, more user-friendly video summaries.
• The MGCNN module provides a robust method for assessing text coherence, contributing to better overall summary generation.