Should robots display what they hear? Mishearing as a practical accomplishment

Area of Science:

• Human-Robot Interaction (HRI)
• Transparency in AI
• Communication Studies

Background:

• Understanding failures in human-robot interaction (HRI) is crucial for designing effective conversational agents.
• The role of transparency, specifically displaying automatic speech recognition (ASR) results, in shaping interaction dynamics is under-explored.

Purpose of the Study:

• To investigate if publicly displaying a robot's ASR results influences the emergence and management of miscommunications in HRI.
• To analyze how the "informational ecology" created by ASR transcripts affects participants' interpretation of robot conduct.

Main Methods:

• Preliminary quantitative analysis of participant gaze behavior during human-robot interaction experiments.
• Micro-analytic approach to examine interaction sequences and the interpretation of robot utterances and gestures.
• Case studies focusing on instances where ASR transcripts altered participants' perception of interaction success.

Main Results:

• Displaying ASR transcripts significantly impacted participants' interpretation of robot actions as inadequate or "failed".
• Instances of miscommunication were re-evaluated as problematic only after participants read the ASR transcript.
• The informational ecology created by ASR display modified the definition of "mutual understanding" participants oriented to.

Conclusions:

• Systematically displaying ASR transcripts plays a critical role in how participants interpret co-constructed actions in HRI.
• "Mistakes" and "errors" are interactional accomplishments, not pre-defined technical or social issues.
• The design decision to display ASR transcripts requires careful consideration due to potential unintended consequences on human-AI communication.