Hierarchical Decoding of Perceived Speech From Non-Invasive Brain Recordings

Area of Science:

• Neuroscience
• Computational Linguistics
• Signal Processing

Background:

• Speech perception is a hierarchical process involving multiple brain regions.
• Existing speech decoding methods rarely incorporate this hierarchical nature.
• Non-invasive brain signal analysis (M/EEG) offers insights into speech perception.

Purpose of the Study:

• To propose a novel hierarchical decoding framework for speech perception.
• To integrate diverse speech representations (mel-spectrogram, Wav2vec 2.0, GPT-2) for improved decoding.
• To evaluate the framework's performance against existing methods.

Main Methods:

• Developed ConvConcatNet, a hierarchical decoder using iterative convolution and concatenation.
• Employed contrastive learning to align neural features with speech representations.
• Validated the framework on Chinese MEG (SMN4Lang) and Dutch EEG (SparrKULee) datasets.

Main Results:

• Wav2vec 2.0 representation showed high decoding performance.
• Integrating all three representations (mel-spectrogram, Wav2vec 2.0, GPT-2) substantially boosted accuracy.
• GPT-2 representation improved decoding for context-dependent words.
• ConvConcatNet achieved state-of-the-art Top-1 accuracy: 35.6% (MEG) and 20.0% (EEG).

Conclusions:

• A hierarchical approach combining diverse speech features enhances non-invasive speech decoding.
• The proposed ConvConcatNet framework effectively extracts and integrates neural patterns.
• This work advances the understanding of speech perception through brain signal decoding.