Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech

Area of Science:

• Neuroscience
• Computer Science
• Biomedical Engineering

Background:

• Brain-computer interfaces (BCIs) aim to decode speech signals using invasive electrocorticogram (ECoG) measurements.
• Decoding imagined (covert) speech presents a significant challenge compared to overt or perceived speech.

Purpose of the Study:

• To investigate the feasibility of decoding Japanese sentences from covert speech using a Transformer neural network.
• To evaluate if ECoG data from overt speech can be used to train a model for decoding covert speech.

Main Methods:

• ECoG data were recorded from 16 epilepsy patients during overt and covert speech of Japanese sentences.
• A Transformer neural network was employed to decode text sentences from covert speech.
• The model was trained and tested using both covert speech data and overt speech data for training covert speech decoding.

Main Results:

• The Transformer model trained on covert speech achieved a 46.6% token error rate (TER) for decoding covert speech.
• A model trained on overt speech data achieved a comparable TER of 46.3% for decoding covert speech.
• These results indicate that overt speech data can effectively train models for covert speech decoding.

Conclusions:

• The challenge of acquiring sufficient training data for covert speech BCI can be mitigated by utilizing readily available overt speech data.
• Employing overt speech data for training enhances the performance of covert speech decoding models.
• This approach offers a practical solution for advancing speech BCI technology, particularly for silent communication applications.