Related Experiment Video
Updated: Sep 18, 2025
12:39
A Methodology for Capturing Joint Visual Attention Using Mobile Eye-Trackers
Published on: January 18, 2020
7.8K
Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics
Jacob P Momsen1,2, Seana Coulson1,3
1San Diego State University and UC San Diego.
Journal of Cognitive Neuroscience
|June 24, 2025
Summary
Co-speech gestures enhance brain activity tracking of speech. This study shows that the movement patterns in gestures, even when simplified, improve how the brain processes spoken language, demonstrating multisensory integration.
Area of Science:
- Neuroscience
- Cognitive Science
- Speech Processing
Background:
- Multisensory integration enhances sensory processing, but its effect on speech and co-speech gestures is not fully understood.
- Previous research focused on visual articulation, leaving the integration of speech with bodily gestures less explored.
Purpose of the Study:
- To investigate the multisensory integration of speech and co-speech gestures.
- To determine if the biological motion of gestures enhances neural tracking of the speech envelope.
Main Methods:
- Recorded EEG from 20 healthy adults watching multimodal speech and gesture clips.
- Used minimalistic "point-light" depictions of speakers to isolate gesture kinematics.
- Employed neural decoder models to predict speech envelope amplitude from EEG data.
Main Results:
- Decoder performance was significantly better for congruent audiovisual (AV) stimuli compared to audio-only or additive models.
- Neural activity showed signatures of multisensory integration between 140-160 milliseconds after speech envelope changes.
- Kinematics of co-speech gestures were sufficient to enhance speech tracking.
Conclusions:
- Co-speech gestures, through their movement patterns, enhance the brain's continuous tracking of speech.
- This study provides novel evidence for super-additive multisensory enhancement in speech processing driven by gesture kinematics.
