A Vision Transformer Architecture For Overt Speech Decoding From ECoG Data

Speech Brain-Computer Interfaces rely on decoding algorithms that transform neural activity into speech. A current challenge is to achieve intelligible speech synthesis in real time from continuous ongoing brain activity, ideally without the need of language models that prevent free-speech production. As a first step toward this goal, we introduce here an encoder-decoder architecture, in which neural data is first encoded into a latent space using a multi-layer vision transformer (ViT), and then these latent variables are converted into acoustic coefficients using a bidirectional LSTM recurrent network. This network is compared to a more conventional architecture where the encoding is performed using a convolutional neural network. Moreover, we introduce a new data-driven data augmentation strategy based on Dynamic Time Warping (DTW) to increase a training dataset based on the intrinsic variability of its input neural features. On two ECoG datasets obtained in participants performing an overt speech task, we found that ViT-encoding outperforms CNN-encoding to predict produced speech offline and that DTW-based data augmentation also improves decoding performance.