Gaussian process inference modelling of dynamic robot control for expressive piano playing

Area of Science:

• Robotics
• Human-Computer Interaction
• Music Technology

Background:

• Playing the piano requires extensive human dexterity and adaptability.
• Advanced musical pieces rely on nuanced, dynamic interactions for delicate expressions.
• The relationship between motor control and sound production in piano playing is hypothesized to be highly nonlinear.

Purpose of the Study:

• To investigate the complex dynamics of human-piano interaction.
• To understand the physical dexterity and adaptability involved in piano playing.
• To explore the nonlinear relationship between key-pressing motor control and resulting sound.

Main Methods:

• Utilized a robotic arm with an elastic finger to perform 3125 key-presses on a digital piano.
• Employed Gaussian Process (GP) inference modeling to analyze data across 10 Musical Instrument Digital Interface (MIDI) playing styles.
• Systematically varied control parameters to investigate motor control and sound output.

Main Results:

• Confirmed a highly nonlinear relationship between robot control parameters and generated piano sounds, particularly for rich expressions and sound dynamics.
• Demonstrated that Gaussian Process models effectively captured this nonlinearity, outperforming traditional linear regression models.
• Found that the robot controller could replicate human-like expressions in some instances, while others require further research.

Conclusions:

• The embodied interaction between a human and a piano exhibits significant nonlinearity, especially in expressive playing.
• Gaussian Process modeling provides a powerful tool for understanding and potentially replicating complex musical expressions.
• Robotic systems show promise in emulating human piano-playing nuances, though further development is needed for certain expressive qualities.