Visual Reasoning: From State to Transformation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Cognitive Science

Background:

• Existing visual reasoning tasks primarily focus on static image analysis, neglecting the crucial aspect of transformation dynamics.
• Human cognition, as described by Piaget's theory, emphasizes inferring dynamics between states, a capability lacking in current AI.
• State-driven visual reasoning has limitations in capturing the complexities of real-world dynamic environments.

Purpose of the Study:

• To introduce a novel transformation-driven visual reasoning (TVR) task that infers intermediate transformations from initial and final states.
• To develop new datasets (TRANCE and TRANCO) to facilitate research in TVR, covering synthetic and real-world scenarios.
• To evaluate the performance of state-of-the-art visual reasoning models on the proposed TVR task.

Main Methods:

• Construction of the TRANCE dataset based on CLEVR, featuring Basic, Event, and View transformation levels.
• Creation of the TRANCO dataset using COIN to incorporate real-world transformation diversity.
• Proposal of a three-stage reasoning framework, TranNet (observing, analyzing, concluding), for TVR evaluation.

Main Results:

• State-of-the-art models achieved good performance on the Basic TVR setting.
• Significant performance gaps were observed between AI models and human-level intelligence on Event, View, and TRANCO datasets.
• Current advanced techniques demonstrate limitations in handling complex, multi-step, and view-variant transformations.

Conclusions:

• The proposed transformation-driven visual reasoning paradigm is crucial for advancing machine visual reasoning capabilities.
• Further research is needed to develop more sophisticated methods to address the challenges in complex visual transformations.
• The new datasets and task benchmark progress in AI's ability to understand dynamic visual environments.