A multi-scale self-supervised hypergraph contrastive learning framework for video question answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Video question answering (VideoQA) requires understanding multimodal information and object interactions.
• Existing Graph Neural Network (GNN) models for VideoQA struggle with capturing high-order relations and leveraging self-supervised signals.

Purpose of the Study:

• To propose a novel Multi-scale Self-supervised Hypergraph Contrastive Learning (MSHCL) framework for enhanced VideoQA.
• To address limitations of existing GNN-based methods in capturing complex, high-order object relationships and utilizing self-supervised learning signals.

Main Methods:

• Constructing a multi-scale temporal-spatial hypergraph to directly model high-order object relations using appearance and motion hyperedges.
• Integrating hypergraph convolution features with a Transformer for global sequence information capture.
• Employing a self-supervised hypergraph contrastive learning task with data augmentation and a question-guided multimodal interaction module.

Main Results:

• The proposed MSHCL framework demonstrates superior performance compared to state-of-the-art methods on three benchmark VideoQA datasets.
• The model effectively captures high-order relations among multiple objects, overcoming limitations of traditional GNNs.
• Self-supervised learning signals within the hypergraph structure significantly enhance accuracy and robustness.

Conclusions:

• The MSHCL framework offers a more effective approach to VideoQA by directly modeling high-order relations and utilizing multi-scale self-supervised learning.
• This method advances video understanding by improving the capture of complex temporal-spatial interactions and object semantics.
• The findings suggest a promising direction for future research in multimodal understanding and question answering.