Using Automatic Speech Recognition to Measure the Intelligibility of Speech Synthesized from Brain Signals

Area of Science:

• Neuroscience
• Artificial Intelligence
• Speech Technology

Background:

• Brain-computer interfaces (BCIs) offer potential for restoring function in neurological injury patients.
• Speech restoration via BCIs synthesizes speech from neural signals of non-speaking individuals.
• Current methods for quantifying BCI speech quality lack objectivity, scalability, and applicability to anarthric users.

Purpose of the Study:

• To introduce a deep learning-based "AI Listener" for objective, rapid, and automatic evaluation of BCI-synthesized speech intelligibility.
• To address the absence of a gold-standard metric for BCI speech quality assessment.

Main Methods:

• Adapted leading Automatic Speech Recognition (ASR) deep learning models (Deepspeech, Wav2vec 2.0, Kaldi) for BCI speech evaluation.
• Evaluated ASR model performance on diverse speech datasets: healthy, dysarthric, and BCI-synthesized speech.
• Utilized XLSR-Wav2vec 2.0, trained on phonemes, for superior speech transcription accuracy.

Main Results:

• The AI Listener demonstrates objective and rapid assessment of BCI speech intelligibility.
• XLSR-Wav2vec 2.0 achieved superior performance in speech transcription accuracy.
• The AI Listener identified several previously published BCI speech datasets as unintelligible, aligning with human listener assessments.

Conclusions:

• The AI Listener provides a robust, scalable, and objective method for evaluating BCI speech.
• This tool can serve as a cost function for improving BCI speech decoding algorithms.
• Findings highlight the potential of AI in advancing speech restoration technologies for individuals with speech impairments.