Decoding EEG Brain Activity for Multi-Modal Natural Language Processing.

Area of Science:

• Neuroscience
• Machine Learning
• Computational Linguistics

Background:

• Human behavioral data from reading, primarily studied for cognitive insights, holds untapped potential for machine learning.
• Electroencephalography (EEG) brain activity, a rich source of human language processing signals, remains largely unexplored in natural language processing (NLP).

Purpose of the Study:

• To systematically analyze the potential of EEG brain activity data for improving NLP tasks.
• To identify the most beneficial EEG signal features for NLP applications.
• To develop and evaluate a multi-modal machine learning architecture integrating textual and EEG data.

Main Methods:

• A large-scale study analyzing EEG brain activity and textual data using a multi-modal machine learning architecture.
• Systematic comparison of broadband EEG signals versus frequency-filtered EEG signals.
• Evaluation of EEG data's impact on sentiment classification and relation detection tasks with various word embedding types.

Main Results:

• Filtering EEG signals into frequency bands significantly improves performance compared to using the broadband signal.
• EEG data enhances binary and ternary sentiment classification, outperforming multiple baselines across various word embedding types.
• For complex tasks like relation detection, EEG data shows promise primarily when combined with contextualized BERT embeddings, indicating areas for further research.
• EEG data demonstrates particular effectiveness in scenarios with limited training data.

Conclusions:

• EEG brain activity, particularly when frequency-filtered, offers a valuable data source for enhancing NLP tasks, especially in low-data regimes.
• The integration of EEG data with advanced NLP models like BERT can lead to improved performance on complex language understanding tasks.
• Further research is warranted to fully leverage the potential of EEG signals in diverse NLP applications.