Everyday Activity Science and Engineering Table Setting Dataset

Understanding human everyday activity planning and execution is crucial to inform cognition-enabled robots and systems. In this paper, we describe the design, collection, validation, and dissemination of the Everyday Activity Science and Engineering Table Setting Dataset (EASE-TSD). EASE-TSD is a dataset of multimodal high-dimensional biosignals synchronously recorded from human subjects who are setting a table in a controlled laboratory setup. Data from 78 sessions are available, each recorded during six table-setting trials in which we capture the planning and execution of human behavior using eight synchronized biosignal streams: marker-based motion capturing, environmental and first-person video cameras, eye-tracking, electromyography, electrodermal activity, acceleration, microphones, and electroencephalography. Participants were instructed to think aloud concurrently and retrospectively to explain and comment on their table-setting actions and the corresponding cognitive processes. EASE-TSD is annotated with a 3-level annotation schema containing phases, activities, motions, and interacted objects. Additionally, the think-aloud (TA) protocols are annotated using TA codes. After recording, the EASE-TSD data undergo semi-automatic labeling, post-processing, and analysis procedures, leveraging latest biosignal processing and machine learning methods.