A Comprehensive Behavioral Dataset for the Abstraction and Reasoning Corpus.

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Human-Computer Interaction

Background:

• The Abstraction and Reasoning Corpus (ARC) is a key benchmark for evaluating machine out-of-distribution generalization.
• Comparing AI performance to human capabilities is crucial for measuring progress in AI research.

Purpose of the Study:

• To introduce H-ARC, a large-scale dataset of human performance on ARC tasks.
• To provide a valuable resource for understanding human abstraction and reasoning mechanisms.
• To inform the development of more human-like AI algorithms.

Main Methods:

• Collected solution attempts from over 1700 humans across all 800 ARC training and evaluation tasks.
• Recorded step-by-step behavioral action traces from the ARC user interface.
• Gathered natural-language descriptions of inferred programs or rules.

Main Results:

• The H-ARC dataset is the largest human evaluation dataset for the ARC benchmark to date.
• The dataset includes human responses, behavioral traces, and solution descriptions for each task.
• This comprehensive dataset enables direct comparison between human and AI performance on visual reasoning.

Conclusions:

• The H-ARC dataset offers significant potential for cognitive science research into human abstraction and reasoning.
• Insights from H-ARC can guide the design of more efficient and human-like AI systems.
• This resource facilitates interdisciplinary research bridging AI and cognitive science.