Investigating joint attention mechanisms through spoken human-robot interaction

Area of Science:

• Cognitive Science
• Human-Robot Interaction
• Psycholinguistics

Background:

• Referential gaze is closely linked to spoken language production and comprehension.
• Listeners can use a speaker's visual attention to aid understanding in shared environments.
• Investigating gaze-following dynamics is crucial for understanding situated language processing.

Purpose of the Study:

• To examine how referential gaze influences utterance comprehension in a controlled human-robot interaction setting.
• To test the hypothesis that gaze cues signal referential intentions, affecting reference resolution.
• To explore the incremental integration of speech and gaze movement during comprehension.

Main Methods:

• Utilized a human-robot interaction paradigm with eye-tracking technology.
• Participants viewed videos of a robot describing objects in a scene while its gaze was manipulated.
• Conducted two eye-tracking experiments to quantify the effects of gaze on comprehension.

Main Results:

• Demonstrated a spectrum of gaze effects, showing both benefits and disruptions to utterance comprehension.
• Found that gaze is used early, even during initial movement, to narrow down potential referents.
• Quantified the impact of gaze cues on the speed and accuracy of reference resolution.

Conclusions:

• Referential gaze from artificial agents influences how humans comprehend spoken language.
• People interpret robot gaze similarly to human gaze, validating human-robot interaction for studying joint attention.
• Findings support the incremental processing of multimodal cues (speech and gaze) in communication.