Whisper automatic speech recognition and humans are similarly sensitive to accents but use sentence context...

Despite advancements in speech recognition technology, questions remain about model generalizability and how many models mirror human recognition. These questions are addressed by comparing OpenAI's Whisper model and 75 human transcribers on 300 English sentences (20 speakers, half female, half male, half US-accented, half Spanish-accented). Sentences ended in 100 target words, each occurring in one sensical sentence (The monkey ate the banana), one nonsensical (The monkey ate the shark), and one nonsensical + nonword (The runkey ate the shark). Target word transcription accuracy was examined both in sentences and in isolation (recordings excised from same sentences). Human transcriptions replicated existing work. There were two major findings. First, the model transcribed Spanish-accented speech less well, and it benefited from sensical contexts, particularly for Spanish-accented speech. Second, sentence context affected models differently than humans: Humans benefited only from sensical sentences (with final words in-sentence more accurate than in isolation), while Whisper benefited from any sentence context. Humans and Whisper showed similar patterns of talker difficulty but different error patterns on isolated words, possibly due to Whisper's word-prediction model being overly restrictive in short utterances. Suggestions for ASR development and the possible practical and scientific contributions of automatic speech recognition to understanding human accent processing are discussed.