Music, body, and machine: gesture-based synchronization in human-robot musical interaction

Area of Science:

• Human-robot interaction
• Robotics in music
• Nonverbal communication in music

Background:

• Human musicians use nonverbal cues for communication during performances.
• Robotic musicians can leverage physical gestures for similar communicative functions.
• Existing research lacks a framework for classifying musical gestures and their impact on human-robot synchronization.

Purpose of the Study:

• To introduce a theoretical framework for classifying musical gestures.
• To evaluate the impact of robotic gestures on human-robot synchronization.
• To assess the influence of Shimon's ancillary and social gestures on synchronization and user experience.

Main Methods:

• Developed a theoretical framework to classify musical gestures.
• Conducted a study with human pianists playing alongside Shimon, a robotic marimba player.
• Assessed participants' synchronization accuracy with tempo changes communicated via Shimon's gestures.

Main Results:

• Robotic non-instrumental gestures significantly improved human-robot synchronization.
• Ancillary and social gestures aided in anticipating tempo and beat changes.
• Participants reported more positive feelings during interactions involving these gestures.

Conclusions:

• Non-music-making gestures are crucial for effective human-robot musical collaboration.
• Robotic gestures enhance anticipation, coordination, and the overall engagement of musical experiences.
• Integrating expressive gestures into robotic musicians can lead to more intuitive and enjoyable human-robot musical partnerships.