SACM: SEEG-Audio Contrastive Matching for Chinese Speech Decoding.

Area of Science:

• Neuroscience
• Biomedical Engineering
• Speech Science

Background:

• Speech disorders like dysarthria and anarthria significantly impact verbal communication.
• Brain-computer interfaces (BCIs) offer a promising avenue for restoring speech through direct neural decoding.

Purpose of the Study:

• To develop and evaluate a speech decoding BCI for Mandarin Chinese.
• To propose a novel contrastive learning algorithm for enhanced speech decoding.

Main Methods:

• Collected stereo-electroencephalography (SEEG) and audio data from ten epilepsy patients during a word-level reading task.
• Developed the SEEG and Audio Contrastive Matching (SACM) algorithm, a contrastive learning approach.
• Leveraged cross-modal correlations between neural activity and audio signals for speech segment decoding.

Main Results:

• The SACM framework significantly surpassed random matching accuracy in both isolated and continuous speech decoding.
• Outperformed SEEG-only baseline models across seven architectures in isolated-word decoding.
• Demonstrated that a single ventral sensorimotor cortex electrode achieved performance comparable to a full electrode array.

Conclusions:

• This work represents the first multimodal decoding approach for tonal speech BCIs.
• The findings highlight the potential of SACM for developing effective speech neuroprostheses.