VideoABC: A Real-World Video Dataset for Abductive Visual Reasoning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Abductive visual reasoning (AVR) traditionally uses static images.
• Instructional videos offer rich, dynamic data for complex reasoning.
• Existing methods lack robust long-term temporal reasoning capabilities.

Purpose of the Study:

• To investigate abductive visual reasoning (AVR) using instructional videos.
• To introduce novel tasks (AVR and AVR++) for evaluating video-based reasoning.
• To develop a model capable of human-level temporal reasoning in visual contexts.

Main Methods:

• Conceptualized two AVR tasks: inferring steps to a goal and explaining implausible options (AVR++).
• Created the VideoABC dataset with 13,526 abductive reasoning questions from 11,827 instructional videos.
• Proposed a Hierarchical Dual Reasoning Network (HDRNet) for capturing long-term dependencies.

Main Results:

• The HDRNet achieved state-of-the-art results on AVR (~74%) and AVR++ (~45%).
• Human performance exceeded 90% accuracy on both tasks, indicating a significant gap.
• The VideoABC dataset and adversarial hypothesis mining facilitated effective problem generation.

Conclusions:

• Current video understanding models struggle with complex temporal reasoning required for AVR.
• The developed HDRNet and VideoABC dataset provide a strong benchmark for future research.
• Significant advancements are needed to bridge the performance gap between models and humans in video-based reasoning.