Reconstructing fine-grained cognition from brain activity.

Area of Science:

• Cognitive Science
• Neuroscience
• Computational Modeling

Background:

• Reconstructing cognitive processes from neural data is challenging, especially in dynamic, interdependent environments like video games.
• Existing methods struggle to capture the fine-grained details of cognitive sequences in high-variability tasks.

Purpose of the Study:

• To introduce and validate the Sketch-and-Stitch method for integrating cognitive models with electroencephalography (EEG) data.
• To reconstruct subject cognition and generate detailed game replays from EEG signals.

Main Methods:

• The Sketch-and-Stitch method employs a two-level approach: 'Sketch' identifies critical events using robust EEG signals, and 'Stitch' fills in detailed actions.
• A cognitive model provides transition probabilities and interval distributions between critical events, integrated into a hidden semi-Markov model.
• Action sequences are selected from a library to best match weaker EEG signal aspects, refining the reconstruction.

Main Results:

• The hidden semi-Markov model successfully identified the most probable sequence and timing of critical events from EEG data.
• The Stitch level effectively selected detailed actions, producing compelling and accurate game replays.
• The method demonstrated efficacy in a highly interdependent and variable video game context.

Conclusions:

• The Sketch-and-Stitch method offers a novel and effective way to reconstruct cognitive processes using EEG and computational models.
• This approach has significant potential for understanding human cognition in complex, dynamic environments.
• The ability to generate detailed replays from EEG opens new avenues for research and application.