Towards decoding individual words from non-invasive brain recordings

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computational Linguistics

Background:

• Deep learning has advanced brain-computer interfaces (BCIs) for language decoding using invasive intracranial recordings.
• Decoding natural language from non-invasive brain signals (EEG, MEG) remains a significant challenge.

Purpose of the Study:

• To introduce and evaluate a deep learning pipeline for decoding individual words from electroencephalography (EEG) and magnetoencephalography (MEG) signals.
• To assess the model's performance across diverse datasets, languages, and experimental conditions.

Main Methods:

• A novel deep learning pipeline was developed for word decoding from EEG and MEG data.
• The pipeline was validated on seven public and two newly collected datasets, totaling 723 participants and over five million words in three languages.
• Performance was evaluated across different recording devices, tasks (reading vs. listening), and data volumes.

Main Results:

• The proposed deep learning model consistently outperformed existing methods in word decoding accuracy across participants, devices, languages, and tasks.
• The model demonstrated the ability to decode words not present in the training set (zero-shot decoding).
• Decoding performance was influenced by the recording modality (MEG superior to EEG) and task (reading superior to listening), and improved with increased training data and testing signal averaging.

Conclusions:

• The developed deep learning pipeline represents a significant advancement in non-invasive natural language decoding from brain activity.
• MEG and reading tasks offer more favorable conditions for decoding compared to EEG and listening.
• Further research and data are crucial for refining non-invasive brain decoders for natural language applications.