Outcome measures based on classification performance fail to predict the intelligibility of binary-masked speech

Area of Science:

• Speech processing
• Auditory perception
• Signal processing

Background:

• The hit-FA (H-FA) metric, a common measure for binary mask estimation, treats mask units independently.
• Error distribution significantly impacts speech intelligibility, a factor not fully captured by H-FA.
• Short-time objective intelligibility (STOI) is an alternative metric using reconstructed speech.

Purpose of the Study:

• To investigate the predictive ability of H-FA and STOI for speech intelligibility.
• To analyze how different error distributions in binary masks affect these metrics.
• To determine the limitations of current metrics in evaluating mask estimation algorithms.

Main Methods:

• Binary mask estimation algorithms were used to generate masks with varying error distributions.
• Speech intelligibility was assessed behaviorally using these masks.
• The performance of H-FA and STOI was compared against behavioral intelligibility scores.

Main Results:

• H-FA demonstrated an inability to accurately predict behavioral intelligibility.
• STOI also showed limitations in predicting intelligibility across different error distributions.
• The study highlights the critical role of error distribution in speech intelligibility.

Conclusions:

• Neither H-FA nor STOI are sufficient on their own for evaluating binary mask estimation algorithms.
• Future evaluations must incorporate the impact of error distribution.
• Relying solely on H-FA or STOI can lead to inaccurate performance assessments.