SACM: SEEG-Audio Contrastive Matching for Chinese Speech Decoding

Speech disorders such as dysarthria and anarthria can severely impair patients' ability to communicate verbally. Speech decoding brain-computer interfaces (BCIs) offer a potential alternative by directly translating speech intentions into spoken words, serving as speech neuroprostheses.

This paper reports an experimental protocol for Mandarin Chinese speech decoding BCIs and proposes a contrastive learning-based decoding algorithm termed SEEG and Audio Contrastive Matching (SACM). Stereo-electroencephalography (SEEG) and synchronized audio data were collected from ten patients with drug-resistant epilepsy as they performed a word-level reading task. SACM leverages the cross-modal correlation between neural activity and audio signals to decode matched speech segments.

The proposed framework achieved accuracies significantly exceeding random matching in both isolated-word and continuous speech decoding tasks, and outperformed SEEG-only baselines across seven backbone architectures in the isolated-word setting. Electrode- wise analysis revealed that a single ventral sensorimotor cortex electrode achieved performance comparable to that of the full electrode array. Our code is publicly available.

To our knowledge, this is the first work on multimodal decoding for tonal speech BCIs.