Multi-Granularity Contrastive Cross-Modal Collaborative Generation for End-to-End Long-Term Video Question Answering

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Long-term Video Question Answering (VideoQA) is complex, requiring semantic understanding of untrimmed videos and cross-modal reasoning.
• Existing methods use feature extractors, leading to domain-independent representations and gradient blocking issues.
• While video-language pre-training models offer end-to-end solutions, they lack domain-specific reasoning and have task formulation disparities.

Purpose of the Study:

• To introduce an end-to-end solution for long-term VideoQA that addresses limitations of current approaches.
• To derive discriminative, concept-rich representations for visual understanding.
• To reformulate VideoQA as a generative task for improved cross-modal fusion and answer generation.

Main Methods:

• Proposed the Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model.
• Introduced Joint Unimodal Modeling (JUM) on a clip-bone architecture for discriminative representations.
• Leveraged Multi-granularity Contrastive Learning (MCL) to capture semantic correspondences.
• Developed a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task.

Main Results:

• The proposed MCG model demonstrates superior performance on six publicly available VideoQA datasets.
• The JUM and MCL components effectively derive discriminative representations with high visual concept relevance.
• The CCG module successfully enables high-semantic fusion and generation for rationalizing and answering questions.

Conclusions:

• The MCG model offers a significant advancement in end-to-end long-term VideoQA.
• The integration of JUM, MCL, and CCG effectively addresses challenges in representation learning and task formulation.
• The generative approach for VideoQA shows strong potential for future research and applications.