Exploring Practical Metrics to Support Automatic Speech Recognition Evaluations.

Area of Science:

• Speech Technology
• Human-Computer Interaction
• Machine Learning

Background:

• Traditional word error rate (WER) metrics for automatic speech recognition (ASR) do not adequately capture error types or reasons, hindering developer feedback.
• In academic settings, ASR errors in transcriptions and captions negatively impact comprehension, especially for students with disabilities, due to domain-specific language and non-native speakers.
• Existing ASR struggles with challenges like noisy environments and specialized terminology common in academic lectures.

Purpose of the Study:

• To discuss the use of additional metrics beyond word error rate for evaluating ASR output quality.
• To explore how these metrics can provide better feedback for machine learning processes in ASR.
• To promote more inclusive practices in virtual conferencing systems through improved ASR.

Main Methods:

• Examining the limitations of word error rate in assessing ASR accuracy for transcriptions and captions.
• Investigating the incorporation of paralinguistic features (timing, intonation, voice quality, speech understanding) into ASR evaluation.
• Analyzing feedback mechanisms to enhance the machine learning process for ASR systems.

Main Results:

• Word error rate provides limited insight into the nature and cause of ASR errors.
• Paralinguistic features offer a richer understanding of speech nuances crucial for academic ASR.
• Integrating new metrics can lead to more accurate and context-aware ASR outputs.

Conclusions:

• Enhanced ASR evaluation requires metrics that go beyond simple word accuracy.
• Incorporating paralinguistic analysis improves ASR performance in complex academic environments.
• Advanced ASR evaluation fosters greater inclusivity and accessibility in virtual communication platforms.