Compositional Attention Networks with Two-Stream Fusion for Video Question Answering

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Video Question Answering (VideoQA) requires effective video representation and spatiotemporal reasoning.
• Existing frameworks often rely on a discriminative video encoder and a question-guided decoder.
• Integrating visual and textual information is crucial for accurate VideoQA.

Purpose of the Study:

• To propose a novel framework, Compositional Attention Networks (CAN), for Video Question Answering.
• To enhance video representation through a two-stream fusion mechanism.
• To improve spatiotemporal reasoning capabilities in VideoQA models.

Main Methods:

• Developed a two-stream encoder sampling video snippets via uniform sampling and action pooling.
• Introduced a compositional attention module for integrating two-stream features.
• Proposed five variants of the compositional attention module with different fusion strategies.

Main Results:

• Achieved new state-of-the-art results on the ActivityNet-QA dataset.
• Demonstrated superior performance on the MSRVTT-QA and MSVD-QA datasets.
• The proposed CAN model effectively integrates multi-stream video features for improved VideoQA.

Conclusions:

• Compositional Attention Networks (CAN) provide a powerful approach for Video Question Answering.
• The two-stream fusion and compositional attention mechanisms are key to CAN's success.
• CAN sets a new benchmark for performance in VideoQA tasks across different dataset types.