Lightweight dense video captioning with cross-modal attention and knowledge-enhanced unbiased scene graph

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Dense video captioning (DVC) traditionally focuses on visual features, often neglecting audio cues.
• This oversight leads to inaccuracies in locating events within video scenes.
• Existing DVC models struggle with overlapping events and logical caption coherence.

Purpose of the Study:

• To develop a novel DVC model (CMCR) that effectively integrates multi-modal information (audio-visual).
• To improve the accuracy of scene event localization and caption generation.
• To enhance the logical consistency and reasoning capabilities of video descriptions.

Main Methods:

• Proposed a Cross-Modal processing (CM) module using cross-modal attention for multi-modal feature encoding.
• Introduced an event refactoring algorithm to address inaccurate localization of overlapping events.
• Incorporated a Commonsense Reasoning (CR) module with a knowledge-enhanced scene graph for improved caption logic.

Main Results:

• The CMCR model demonstrated superior performance compared to state-of-the-art methods on the ActivityNet Captions dataset.
• Ablation studies confirmed the significant contributions of both the CM and CR modules.
• The model achieved more accurate event localization and logically coherent captions.

Conclusions:

• Integrating audio-visual information and commonsense reasoning significantly enhances dense video captioning.
• The proposed CMCR model offers a robust solution for accurate and contextually relevant video scene descriptions.
• Future work can explore further refinements in multi-modal fusion and knowledge integration for DVC.