Time-frequency scattering accurately models auditory similarities between instrumental playing techniques.

Area of Science:

• Music Information Retrieval
• Computational Auditory Perception
• Machine Listening

Background:

• Traditional music similarity retrieval often simplifies timbre, using instrument identity as a proxy for quality.
• Existing methods struggle to capture the nuances of playing techniques (e.g., vibratos, glissandos) and individual perceptual differences.
• Timbre perception is crucial for musical expressivity in diverse musical contexts.

Purpose of the Study:

• To investigate the flexibility of timbre perception beyond instrument and playing technique classifications.
• To develop a machine listening model for recovering auditory similarity graphs across various instruments and techniques.
• To enhance music similarity retrieval systems by incorporating nuanced timbre information.

Main Methods:

• Human participants (N=31) clustered 78 isolated musical notes based on perceived timbre.
• A machine listening model was developed using joint time-frequency scattering features for spectrotemporal modulations.
• The model employed the large-margin nearest neighbor (LMNN) algorithm to minimize triplet loss in the timbre cluster graph.

Main Results:

• Human perception of timbre suggests a more flexible taxonomy than instrument or playing technique alone.
• The proposed machine listening model achieved a state-of-the-art average precision at rank five (AP@5) of 99.0% ±1 over 9346 isolated notes.
• Ablation studies confirmed the critical contribution of both the scattering transform and LMNN algorithm to model performance.

Conclusions:

• Timbre perception is multifaceted and not solely defined by instrument or playing technique.
• The developed machine listening model effectively captures complex auditory similarities, advancing music information retrieval.
• This approach offers a more personalized and accurate method for understanding and retrieving music based on timbre.