Moving beyond word error rate to evaluate automatic speech recognition in clinical samples: Lessons from research...

Area of Science:

• Computational Linguistics
• Psychiatry
• Artificial Intelligence in Healthcare

Background:

• Automatic Speech Recognition (ASR) is essential for Natural Language Processing (NLP) in mental health research, enabling large-scale studies and scalable clinical tools.
• Understanding ASR performance in clinical populations is critical for the safe and effective deployment of these technologies in healthcare settings.

Purpose of the Study:

• To evaluate ASR performance on speech samples from individuals diagnosed with schizophrenia-spectrum disorders.
• To identify factors influencing ASR accuracy, such as symptom severity and demographic variables.
• To assess the impact of ASR-generated transcripts on subsequent NLP analyses and their correlation with clinical measures.

Main Methods:

• Analysis of 50 speech samples from individuals with schizophrenia-spectrum disorders.
• Calculation of Word Error Rates (WER) for ASR transcripts.
• Comparison of ASR transcripts with manual transcripts using NLP metrics like GloVe semantic similarity and sentence count.
• Evaluation of correlations between NLP metrics derived from ASR transcripts and clinical symptom scores.

Main Results:

• Word Error Rates (WER) ranged from 0.31 to 0.58, with variations linked to country of birth and positive symptom severity.
• ASR transcripts exhibited higher GloVe semantic similarity and fewer sentences compared to manual transcripts.
• NLP metrics derived from ASR transcripts showed weaker correlations with clinical symptom scores than those from manual transcripts.

Conclusions:

• ASR performance assessment in clinical settings must extend beyond WER to include error type, meaning, and context.
• Differences in ASR transcripts can influence the outcomes of NLP analyses in mental health research.
• The study provides a framework for evaluating ASR in clinical research, guiding future development and implementation in electronic health records, voice chatbots, and clinical decision support systems.