Neural correlates to automatic behavior estimations from RGB-D video in epilepsy unit

Area of Science:

• Neuroscience
• Biomedical Engineering
• Computer Vision

Background:

• Neural monitoring is crucial for understanding brain function, particularly in epilepsy.
• Current methods often lack integration with naturalistic patient behaviors.
• Electrocorticography (ECoG) provides high-resolution neural data but requires correlation with observable actions.

Purpose of the Study:

• To develop and implement a minimally intrusive, multi-modal system for synchronized neural and behavioral data capture.
• To propose an automated approach for functional brain mapping in a clinical setting.
• To correlate electrocorticography (ECoG) signals with naturalistic arm movements.

Main Methods:

• Designed a multi-modal capture system integrating RGB-D audio-video with ECoG recordings.
• Utilized computer vision algorithms to estimate hand and arm movement kinematics from video data.
• Analyzed high-gamma (70-110 Hz) spectral features of ECoG data.
• Applied statistical tests to correlate movement behaviors with neural activity.

Main Results:

• Successfully synchronized audio-video data with ECoG recordings.
• Generated coarse functional maps of ECoG electrodes correlated to contralateral arm movements.
• Demonstrated electrode responses to natural arm movements.

Conclusions:

• The developed system facilitates automated functional brain mapping in clinical settings.
• This approach holds potential for enhancing neural monitoring and brain-computer interfaces.
• Synchronized multi-modal data capture offers a powerful tool for studying brain function during natural behaviors.