A Vision Transformer Architecture For Overt Speech Decoding From ECoG Data.

Area of Science:

• Neuroscience
• Computer Science
• Artificial Intelligence

Background:

• Speech Brain-Computer Interfaces (BCIs) decode neural activity into synthesized speech.
• A key challenge is real-time, intelligible speech synthesis from continuous brain activity without restrictive language models.
• Current BCIs often face limitations in natural, free-speech production.

Purpose of the Study:

• To develop and evaluate a novel encoder-decoder architecture for speech BCI.
• To compare a Vision Transformer (ViT) based encoder against a Convolutional Neural Network (CNN) encoder.
• To introduce and assess a Dynamic Time Warping (DTW) based data augmentation strategy.

Main Methods:

• An encoder-decoder architecture utilizing a multi-layer Vision Transformer (ViT) to encode neural data into a latent space.
• A bidirectional LSTM recurrent network to convert latent variables into acoustic coefficients.
• Comparison with a conventional architecture using a Convolutional Neural Network (CNN) for encoding.
• Implementation of a data-driven data augmentation strategy using Dynamic Time Warping (DTW).

Main Results:

• The Vision Transformer (ViT)-based encoder demonstrated superior performance compared to the Convolutional Neural Network (CNN)-based encoder in predicting produced speech offline.
• The Dynamic Time Warping (DTW)-based data augmentation strategy significantly improved speech decoding performance.
• The proposed architecture shows promise for more natural and intelligible speech synthesis in BCIs.

Conclusions:

• The Vision Transformer (ViT) architecture is effective for encoding neural activity in speech BCIs.
• Dynamic Time Warping (DTW) data augmentation enhances the robustness and accuracy of speech decoding models.
• These advancements represent a significant step towards real-time, natural speech synthesis from brain activity.