Dual-level dynamic heterogeneous graph network for video question answering.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Video Question Answering (VideoQA) is crucial for vision-language understanding.
• Existing VideoQA datasets lack entity and event details, hindering Vision Language Model (VLM) reasoning.
• VLMs often rely on shortcuts or irrelevant visual context due to data limitations.

Purpose of the Study:

• To address limitations in VideoQA datasets by augmenting entity and event information.
• To propose a novel Dual-Level Dynamic Heterogeneous Graph Network (DDHG) for improved VideoQA.
• To enhance multi-modal grounding and reasoning capabilities in VideoQA models.

Main Methods:

• Developed event and entity augmentation strategies to enrich VideoQA datasets.
• Proposed the Dual-Level Dynamic Heterogeneous Graph Network (DDHG) incorporating transformer layers.
• Utilized entity-level and event-level heterogeneous graphs for multi-modal semantic grounding.
• Implemented a Dual-level Cross-modal Interaction Module for feature integration and answer prediction.

Main Results:

• The proposed DDHG method significantly outperforms existing VideoQA models on complex event-based datasets (Causal-VidQA, NExT-QA).
• Demonstrated superior performance in event content prediction compared to state-of-the-art approaches.
• Showcased improved intricate grounding and reasoning among multi-modal entities and events.

Conclusions:

• The data augmentation and DDHG model effectively address challenges in VideoQA.
• The approach enhances the ability of VLMs to understand and reason about complex events in videos.
• This work advances the field of vision-language understanding with a more robust VideoQA solution.