From Word Embedding to Reading Embedding Using Large Language Model, EEG and Eye-tracking.

Area of Science:

• Cognitive Science
• Neuroscience
• Artificial Intelligence

Background:

• Reading comprehension is crucial for learning but challenging for many.
• Current methods for assessing reading comprehension lack precision at the word level.

Purpose of the Study:

• To develop novel Brain-Computer Interface (BCI) tasks for predicting word relevance in reading.
• To integrate Large Language Models (LLMs), electroencephalography (EEG), and eye-tracking for enhanced reading comprehension analysis.

Main Methods:

• Utilized state-of-the-art LLMs to guide a new reading embedding representation.
• Integrated EEG and eye-tracking biomarkers via an attention-based transformer encoder.
• Fine-tuned a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model for word embedding.

Main Results:

• Achieved a mean 5-fold cross-validation accuracy of 68.7% across nine subjects.
• Reached a highest single-subject accuracy of 71.2% in predicting word relevance.
• The fine-tuned BERT model attained 92.7% accuracy for word embedding, validating LLM findings.

Conclusions:

• Pioneered the integration of LLMs, EEG, and eye-tracking for word-level reading comprehension prediction.
• Demonstrated the potential of BCI and AI in developing tools to assist reading.
• The study provides a foundation for future research in neuro-adaptive reading technologies.