Co-Speech Hand Gestures Are Used to Predict Upcoming Meaning

Area of Science:

• Cognitive Science
• Psycholinguistics
• Neuroscience

Background:

• Face-to-face communication integrates speech and gesture.
• The precise mechanisms by which gestures facilitate language comprehension are not fully understood.
• Investigating predictive processing in multimodal communication is crucial.

Purpose of the Study:

• To determine if semantic information from gestures is used for predicting upcoming speech.
• To explore the cognitive and neural underpinnings of gesture-based prediction.
• To enhance theories of linguistic prediction by including non-verbal cues.

Main Methods:

• A Cloze experiment assessed explicit word prediction based on gestures.
• An electroencephalography (EEG) experiment measured brain activity during speech-gesture processing.
• Participants listened to questions from a virtual avatar paired with iconic gestures or control movements.

Main Results:

• Gestures significantly improved participants' ability to predict upcoming words in the Cloze task.
• EEG data revealed reduced alpha and beta power, indicating anticipatory brain activity.
• Gestures led to reduced N400 amplitudes, signifying facilitated semantic processing.

Conclusions:

• Comprehenders actively use semantic information from iconic gestures to predict upcoming meaning.
• Communicative bodily signals serve as predictive cues in face-to-face interaction.
• Linguistic prediction theories must incorporate multimodal signals for a complete understanding of language processing.