Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement

Area of Science:

• Human-Robot Interaction (HRI)
• Behavioral Sciences
• Cognitive Science
• Psychology

Background:

• A confidence crisis exists in scientific disciplines, particularly those studying human behavior, due to failed replication studies.
• Human-Robot Interaction (HRI) research heavily relies on understanding human behavior, cognition, and emotion, making it susceptible to similar issues.

Purpose of the Study:

• To investigate whether HRI user studies are experiencing a replicability crisis.
• To identify the causes and extent of potential replicability issues within HRI research.
• To propose solutions for improving the stability and reliability of future HRI studies.

Main Methods:

• Overview of the replicability crisis in behavioral sciences and its causes.
• Structural comparison of HRI research practices with disciplines facing replicability challenges.
• Systematic review of meta-analyses in HRI user studies to identify factors influencing replicability.
• Summary of existing replication studies in HRI to provide direct evidence.

Main Results:

• HRI user studies exhibit problems mirroring those in behavioral sciences, including small sample sizes, theoretical deficits, and insufficient data reporting.
• These issues contribute to a lack of confidence in the reproducibility of HRI research findings.

Conclusions:

• HRI user studies share significant methodological and reporting challenges with other behavioral sciences, impacting replicability.
• Statistical, methodological, and social reforms are necessary to bolster the stability and credibility of HRI research.
• The article serves as a foundation for dialogue and outlines potential improvements for the HRI field.