Event Graph Guided Compositional Spatial-Temporal Reasoning for Video Question Answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Video question answering (VideoQA) requires complex compositional reasoning across multi-level visual concepts.
• Existing methods using fixed-duration clip features struggle to capture crucial concepts at various granularities.

Purpose of the Study:

• To develop a novel approach for VideoQA that effectively represents and reasons over complex video events.
• To overcome the limitations of fixed-duration clip features in capturing multi-level visual information.

Main Methods:

• Representing videos using a hierarchical Event Graph with nodes for visual concepts (object, relation, scene, action) and edges for spatial-temporal relationships.
• Proposing a Hierarchical Spatial-Temporal Transformer (HSTT) that utilizes the Event Graph for compositional reasoning.
• Employing an improved graph search algorithm for node encoding based on semantic hierarchy and occurrence time.
• Introducing edge-guided attention to integrate spatial-temporal context among nodes.

Main Results:

• The proposed Event Graph and HSTT method significantly outperforms existing VideoQA models on challenging AGQA and STAR datasets.
• The approach demonstrates superior performance even compared to models pre-trained with large-scale external data.
• The hierarchical representation and attention mechanism effectively capture multi-level visual concepts for improved reasoning.

Conclusions:

• The Event Graph representation combined with HSTT offers a powerful framework for VideoQA.
• This method advances the state-of-the-art in compositional reasoning for complex video understanding.
• The proposed approach provides a more effective way to extract and combine visual concepts for accurate video question answering.