Brain2Char: a deep architecture for decoding text from brain recordings.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Biomedical Engineering

Background:

• Brain-computer interfaces (BCIs) offer potential for communication restoration in individuals with severe speech impairments.
• Decoding language representations directly from neural activity is crucial for advancing BCI technology.
• Electrocorticography (ECoG) provides high-resolution brain signals for decoding complex cognitive processes.

Purpose of the Study:

• To introduce Brain2Char, a novel deep network architecture for end-to-end text decoding from ECoG signals.
• To enhance communication capabilities for individuals with neurological conditions affecting speech.
• To establish a high-performance communication BCI system.

Main Methods:

• Utilized a novel deep network architecture, Brain2Char, combining 3D Inception layers, bidirectional recurrent layers, and dilated convolutions.
• Implemented language model-weighted beam search for character sequence decoding and connectionist temporal classification loss.
• Incorporated regularization techniques with auxiliary losses on latent representations for articulatory movements, speech acoustics, and session-specific nonlinearities.

Main Results:

• Brain2Char achieved word error rates of 10.6%, 8.5%, and 7.0% in three out of four participants.
• Performance was evaluated on vocabulary sizes ranging from 1200 to 1900 words.
• Demonstrated successful end-to-end decoding of text from brain signals.

Conclusions:

• The Brain2Char framework represents a significant advancement in decoding text directly from brain signals.
• The developed BCI system shows potential for high-performance communication restoration.
• This approach paves the way for more effective communication tools for individuals with speech disabilities.