Brain-CLIPLM: semantic compression for EEG-to-text decoding

Area of Science:

• Neuroscience
• Cognitive Science
• Artificial Intelligence

Background:

• Decoding natural language from non-invasive electroencephalography (EEG) faces limitations due to low signal-to-noise ratio and restricted information bandwidth.
• The direct recovery of sentence-level language from EEG signals is often considered too demanding given these constraints.

Purpose of the Study:

• To investigate if non-invasive EEG signals preserve recoverable semantic anchors rather than the full lexical-syntactic form of a sentence.
• To propose and evaluate a novel framework, Brain-CLIPLM, for improved EEG-to-text decoding by addressing the mismatch between information scale and decoding granularity.

Main Methods:

• Introduced the semantic compression hypothesis, suggesting EEG captures semantic anchors.
• Developed Brain-CLIPLM, a two-stage framework: Stage 1 recovers ordered semantic anchors using contrastive learning; Stage 2 reconstructs sentences using a retrieval-grounded large language model with chain-of-thought reasoning.
• Utilized the Zurich Cognitive Language Processing (ZuCo) benchmark for evaluation.

Main Results:

• Brain-CLIPLM achieved 67.6% Top-5 and 85.0% Top-25 sentence retrieval accuracy on the ZuCo benchmark.
• Optimal performance was observed at intermediate semantic anchor granularity.
• Control analyses confirmed that EEG-derived anchors contain sentence-specific information beyond language model priors.

Conclusions:

• EEG-to-text decoding is more effectively framed as recovering compressed semantic content before anchor-guided sentence reconstruction.
• The proposed framework aligns decoding complexity with the recoverable neural information scale, improving natural language decoding from EEG.
• Findings suggest that non-invasive EEG may preserve essential semantic information for language understanding.