Brain-to-text decoding with context-aware neural representations and large language models.

Area of Science:

• Neuroscience
• Speech Technology
• Biomedical Engineering

Background:

• Restoring communication for individuals with speech impairments via neural decoding is a key research area.
• Previous methods used phonemes, ignoring context-dependent neural activity, leading to suboptimal speech decoding.
• Context-aware modeling is crucial for improving the accuracy of brain-to-text systems.

Purpose of the Study:

• To propose and evaluate diphones as a context-aware target for decoding neural activity into speech.
• To integrate diphones into existing phoneme decoding frameworks using a novel divide-and-conquer strategy.
• To improve speech decoding performance by leveraging enhanced contextual information from diphones.

Main Methods:

• Developed a novel divide-and-conquer strategy to integrate diphones into phoneme decoding frameworks.
• Modeled phoneme distribution by marginalizing over the diphone distribution for enhanced context.
• Evaluated the approach on the Brain-to-Text 2024 benchmark dataset.

Main Results:

• Achieved a state-of-the-art Phoneme Error Rate (PER) of 15.34%, outperforming the monophone-based decoding PER of 16.62%.
• When combined with fine-tuned Large Language Models (LLMs), the method yielded a Word Error Rate (WER) of 5.77%.
• Significantly outperformed the leading benchmark method's WER of 8.93%.

Conclusions:

• Diphone-based decoding offers a superior, context-aware approach for translating neural activity to speech.
• This method effectively balances contextual richness with manageable complexity for phoneme-to-text conversion.
• The proposed approach represents a significant advancement in brain-computer interfaces for speech restoration.