Dynamic Spatio-Temporal Graph Reasoning for VideoQA With Self-Supervised Event Recognition.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Video question answering (VideoQA) models struggle with complex, multi-event scenarios.
• Current models often rely on global visual features, missing object-level and event-level semantics.
• Existing graph-based methods do not dynamically incorporate question context or event clues.

Purpose of the Study:

• To develop a novel model for VideoQA that addresses limitations in handling complex events.
• To improve the comprehensive understanding of visual content in videos, especially for dynamic interactions.
• To enhance VideoQA performance by better capturing object and event-level semantics.

Main Methods:

• Proposed a Self-supervised Dynamic Graph Reasoning (SDGraphR) model.
• Developed a question-guided spatio-temporal graph to encode object correlations and correspondences.
• Implemented self-supervised learning using an auxiliary event recognition task guided by question cues.

Main Results:

• The SDGraphR model dynamically encodes spatial and temporal object relationships.
• Leveraging question semantics improved the model's ability to recognize implicit event-level clues.
• Achieved substantial improvements over existing VideoQA baselines in extensive experiments.

Conclusions:

• The SDGraphR model offers a significant advancement in VideoQA, particularly for complex event scenarios.
• Dynamic graph reasoning and self-supervised learning are effective strategies for improving VideoQA.
• The approach enhances the model's ability to understand intricate visual narratives in videos without additional annotations.