Decoding of the speech envelope from EEG using the VLAAI deep neural network

Area of Science:

• Neuroscience
• Machine Learning
• Signal Processing

Background:

• Linear models struggle to capture the brain's complex, dynamic nature for speech processing.
• Existing models often require extensive subject-specific training data.

Purpose of the Study:

• Introduce a novel deep learning architecture, the Very Large Augmented Auditory Inference (VLAAI) network, for speech decoding.
• Evaluate the VLAAI network's performance against state-of-the-art models and assess its generalization capabilities.

Main Methods:

• Developed the VLAAI network, a novel deep learning architecture for speech decoding.
• Utilized ablation studies to determine the impact of network components on performance.
• Validated the model on holdout and public unseen datasets, and analyzed the effect of training set size.

Main Results:

• The VLAAI network achieved a median Pearson correlation of 0.19, a 52% improvement over linear models.
• Non-linear components and output context modules were identified as key performance drivers.
• The model demonstrated robust generalization to unseen subjects and stimuli, outperforming baselines on public data.
• Performance scaled with training data size following a hyperbolic tangent function.
• Subject-specific fine-tuning with minimal data (≥5 minutes) yielded up to a 34% performance increase.

Conclusions:

• The VLAAI network offers a significant advancement in subject-independent speech decoding from brain signals.
• The architecture's non-linear components and context module are crucial for its effectiveness.
• VLAAI shows excellent generalization and potential for subject-specific adaptation, requiring limited data.