Generative language reconstruction from brain recordings.

Area of Science:

• Neuroscience
• Cognitive Science
• Artificial Intelligence

Background:

• Reconstructing language from non-invasive brain recordings is a significant challenge.
• Previous methods relied on classification with pre-defined language candidates.

Purpose of the Study:

• To develop a new method for language reconstruction using auto-regressive generation with large language models (LLMs).
• To improve the alignment of reconstructed language with original stimuli, especially for surprising content.
• To enable the reconstruction of extended language stimuli from brain recordings.

Main Methods:

• Utilized functional magnetic resonance imaging (fMRI) data.
• Employed an auto-regressive generation approach using LLMs, directly taking decoded brain representations as input.
• Compared the generative method against classification-based approaches.

Main Results:

• The generative method produced language reconstructions more closely aligned with stimuli compared to classification methods.
• The approach successfully reconstructed a 10-minute language stimulus.
• The method demonstrated comparable or superior performance to existing techniques and can estimate the likelihood of semantic content generation.

Conclusions:

• The study validates the effectiveness of brain-language interfaces in a generative framework.
• This approach offers a powerful and efficient way to map functional language representations in the brain.
• The generative setup overcomes limitations of candidate-based classification for brain-based language reconstruction.