An EEG Dataset for Multimodal Semantic Alignment and Neural Decoding during Reading and Listening

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computational Linguistics

Background:

• Large-scale benchmark datasets are crucial for EEG-based neural decoding.
• Paired brain-language data across speaking, listening, and reading modalities are essential for aligning neural activity with large language models (LLMs).
• Such datasets are scarce, particularly for non-English languages.

Purpose of the Study:

• To present ChineseEEG-2, a high-density EEG dataset for benchmarking neural decoding models in real-world Chinese language tasks.
• To enable precise temporal and semantic alignment across Reading Aloud (RA) and Passive Listening (PL) modalities.
• To support joint semantic alignment learning across speaking, listening, and reading, and promote brain-LLM alignment.

Main Methods:

• Building on the previous ChineseEEG dataset, ChineseEEG-2 incorporates active modalities: Reading Aloud (RA) and Passive Listening (PL) using the same Chinese corpus.
• Simultaneous EEG and audio recording during ~10.8 hours of RA from four participants.
• EEG recording during ~21.6 hours of PL from eight participants, using the RA recordings.

Main Results:

• ChineseEEG-2 includes EEG signals, speech audio, aligned semantic embeddings from pre-trained language models, and task labels.
• The dataset enables precise temporal and semantic alignment across RA and PL modalities.
• It supports joint semantic alignment learning across speaking, listening, and reading.

Conclusions:

• ChineseEEG-2 provides a benchmark dataset for next-generation neural semantic decoding, especially in Chinese.
• It enables benchmarking of neural decoding algorithms under multimodal language tasks.
• The dataset promotes brain-LLM alignment in complex language processing scenarios.