Parse, Align and Aggregate: Graph-Driven Compositional Reasoning for Video Question Answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal Large Language Models (MLLMs) in Video Question-Answering (VideoQA) often lack transparency and verifiability in their reasoning processes.
• Existing VideoQA benchmarks primarily focus on final answer accuracy, neglecting the analysis of underlying reasoning steps.

Purpose of the Study:

• To develop a novel framework, QPVA3 (Question Parsing, Video Alignment and Answer Aggregation), for enhancing transparency and verifiability in VideoQA.
• To introduce new metrics for assessing compositional consistency in VideoQA reasoning.
• To create a comprehensive VideoQA benchmark (QPVA3Bench) with detailed reasoning annotations.

Main Methods:

• The QPVA3 framework utilizes a compositional graph to guide visual and logical reasoning, comprising a planner, executor, and reasoner.
• The planner decomposes questions into a compositional graph, the executor aligns video content and answers sub-questions, and the reasoner aggregates answers based on reasoning logic and visual evidence.
• Novel compositional consistency metrics were developed to evaluate the reasoning process.

Main Results:

• The QPVA3 framework demonstrated improved consistency and accuracy over existing baselines on VideoQA tasks.
• The proposed framework leads to a more transparent and verifiable VideoQA system.
• QPVA3Bench provides a valuable resource for evaluating and advancing VideoQA reasoning.

Conclusions:

• The QPVA3 framework offers a significant advancement in creating more transparent and verifiable VideoQA systems.
• The compositional graph-driven approach enhances the interpretability of machine reasoning in complex video content.
• The developed benchmark and metrics facilitate further research into the reasoning capabilities of MLLMs for VideoQA.