Evaluating OpenAI's Whisper ASR: Performance analysis across diverse accents and speaker traits

Area of Science:

• Computational Linguistics
• Speech Processing
• Artificial Intelligence

Background:

• Automatic Speech Recognition (ASR) systems are crucial for human-computer interaction.
• Evaluating ASR performance across diverse linguistic and demographic groups is essential for equitable technology.
• Whisper is a widely used, open-source ASR system with varying performance characteristics.

Purpose of the Study:

• To assess the performance of the Whisper ASR system across various native and non-native English accents.
• To investigate the impact of speaker traits (sex, L1 typology, L2 proficiency) on Whisper's word error rate.
• To compare Whisper's performance on read versus conversational speech.

Main Methods:

• The study analyzed Whisper's word error rate (WER) on a dataset encompassing diverse English accents.
• Statistical analyses were employed to identify correlations between speaker characteristics and ASR accuracy.
• Performance was compared between read and conversational speech samples.

Main Results:

• Whisper demonstrated superior performance on American English accents compared to British and Australian English.
• Native English accents generally yielded higher accuracy than non-native accents.
• Whisper performed better on read speech than conversational speech, with variations influenced by speaker gender.

Conclusions:

• Whisper's ASR accuracy is influenced by English accent, native language status, and speech style.
• Speaker characteristics like sex, L1, and L2 proficiency are associated with word error rates.
• Further research is needed to optimize ASR for diverse user populations and speech contexts.