Contrastive Video Question Answering via Video Graph Transformer

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Video question answering (VideoQA) is a challenging task requiring understanding of complex spatio-temporal relationships.
• Existing VideoQA models often struggle with nuanced reasoning and efficient cross-modal understanding.

Purpose of the Study:

• To propose a novel Video Graph Transformer model (CoVGT) for enhanced VideoQA.
• To improve spatio-temporal reasoning and video-text contrastive learning for VideoQA tasks.

Main Methods:

• Developed a dynamic graph transformer module to encode videos by capturing objects, relations, and dynamics.
• Employed separate video and text transformers with cross-modal interaction modules for contrastive learning.
• Utilized joint fully- and self-supervised contrastive objectives for optimization.

Main Results:

• CoVGT demonstrates superior performance on video reasoning tasks compared to prior art.
• Achieved better results than models pretrained on significantly larger datasets.
• Showcased effectiveness with orders of magnitude less data for cross-modal pretraining.

Conclusions:

• CoVGT offers a highly effective and superior solution for VideoQA.
• The model exhibits potential for more data-efficient pretraining strategies.
• Highlights the advantage of explicit spatio-temporal encoding and contrastive learning in VideoQA.