Bidirectional effects of speech and gesture: evidence from a novel levels-of-congruence paradigm

Previous research has shown that speech and gesture are integrated during comprehension, with differences in response times and electrophysiological activity observed between incongruent and congruent gesture-speech pairings. Beyond these incongruency effects, it remains unclear whether comprehenders use gestures to adapt or extend conventional interpretations for novel communicative purposes.

The current study examined the bidirectional influence of speech and gesture by employing a novel levels-of-congruence paradigm, in which the same metaphor (e.g., Men are oaks) was paired with a gesture depicting either its preferred interpretation (e.g., men are robust) or a less preferred but possible interpretation (e.g., men are tall). In Experiment 1, participants rated the meaningfulness of preferred and possible interpretations across three gesture conditions (gesture-of-preferred, gesture-of-possible, non-gestured). In Experiment 2, participants selected a single interpretation under the same conditions. In Experiment 3, participants viewed clips of an actor producing the same gesture and utterance combination (e.g., flexing both arms while saying Men are oaks) in one of three multimodal conditions: no additional cues (muted, blurred lips), only lip movements (muted, visible lips), or audible speech. Then they rated how well the gesture matched the gestured interpretation (e.g., robust), an alternative interpretation consistent with the speech but not the gesture (e.g., tall), and an unrelated interpretation (e.g., hikers).

In Experiments 1 and 2, participants consistently rated the preferred interpretation as the most meaningful and selected it most often across all gesture conditions. However, gestures depicting the possible interpretation significantly increased both the meaningfulness ratings and the likelihood of being selected of the possible interpretation. In Experiment 3, the presence of speech cues significantly increased ratings for the alternative interpretation compared with the condition without additional cues.

These findings demonstrate bidirectional interactions between speech and gesture during metaphor comprehension. Gestures constrained metaphor interpretation, whereas speech broadened the range of interpretations attributed to gestures. Overall, the results support the integrated-systems hypothesis by showing that speech and gesture jointly contribute to the construction of meaning.