A perceptual similarity space for speech based on self-supervised speech representations.

Area of Science:

• Speech processing
• Acoustic phonetics
• Machine learning

Background:

• Speech recognition systems struggle with non-optimal conditions like background noise and second-language (L2) speech.
• Existing methods analyzing spectro-temporal properties of speech leave much intelligibility variation unexplained.
• A need exists for novel approaches to understand speech variability and improve recognition accuracy.

Purpose of the Study:

• To investigate an alternative approach to speech analysis using perceptual similarity spaces.
• To determine if perceptual similarity can explain intelligibility variations in L1 and L2 speech.
• To assess the potential of self-supervised learning for encoding speech distinctions.

Main Methods:

• Utilized self-supervised learning to create a perceptual similarity space for speech samples.
• Encoded speech distinctions without relying on pre-defined acoustic features or speech-to-text alignment.
• Quantified distances between first-language (L1) and L2 English speech samples in this space.

Main Results:

• L2 English speech samples showed greater variability (less tight clustering) in the perceptual space compared to L1 samples.
• Distances within the perceptual similarity space correlated with human listener performance.
• L1 English listeners exhibited lower recognition accuracy for L2 speakers whose speech was more distant in the space.

Conclusions:

• Perceptual similarity, captured by self-supervised learning, offers a powerful new dimension for speech analysis.
• This approach provides a more comprehensive explanation for intelligibility variations than traditional feature-based methods.
• Perceptual similarity spaces may form the foundation for next-generation speech and language analysis tools.