Decoding Chinese speech across multiple neural conditions via EEG: dataset construction and interpretability driven...

Area of Science:

• Neuroscience
• Artificial Intelligence
• Speech Technology

Background:

• Brain-computer interfaces (BCIs) show potential for speech assistance, but existing datasets primarily use Western languages.
• Chinese Mandarin, a tonal language, has unique speech mechanisms not well-represented in current BCI research datasets.
• This limits the development of BCIs for Mandarin-speaking individuals with speech impairments.

Purpose of the Study:

• To establish a systematic electroencephalogram (EEG) dataset for Chinese Mandarin speech.
• To evaluate speech decoding and classification performance using advanced AI models.
• To explore the potential of BCIs for communication assistance in Mandarin speakers.

Main Methods:

• Collected a novel EEG dataset using Mandarin tonal-vowels and common vocabularies under four conditions (overt, overt-noisy, intend, imagine).
• Employed Short-Time Fourier Transform with Support Vector Machine (STFT-SVM) and the EEG-Conformer model for speech decoding.
• Developed a multi-task EEG-Conformer architecture for unified decoding and cross-condition classification, incorporating Shapley value for model interpretability.

Main Results:

• The EEG-Conformer model achieved significant decoding accuracy (69.83% in normal, 61.46% in simulated disorder conditions).
• Multi-task classification accuracy across conditions exceeded 97%.
• Model interpretability identified key electrodes, improving performance with reduced channel input.

Conclusions:

• The developed Mandarin EEG dataset and models demonstrate the feasibility of neural signal decoding for communication assistance.
• This research highlights the decodability of Chinese Mandarin EEG data, paving the way for future BCI applications.
• Findings offer practical recommendations for designing Chinese BCI systems.