Interpretable Semantic Vectors from a Joint Model of Brain- and Text-Based Meaning

Area of Science:

• Computational linguistics
• Cognitive neuroscience
• Natural language processing

Background:

• Vector space models (VSMs) represent word meanings using high-dimensional spaces, typically derived from large text corpora.
• Existing VSMs capture semantics based on textual context but lack a direct link to cognitive processes.
• Brain activation data offers a complementary source of semantic information, reflecting how humans process word meanings.

Purpose of the Study:

• To introduce a novel algorithm, JNNSE, for creating VSMs that integrates both corpus-derived and brain activation data.
• To develop a more comprehensive and accurate model of word semantics by leveraging complementary data sources.
• To enhance the representation of mental vocabularies by incorporating neuroimaging insights.

Main Methods:

• Developed the Joint Neural Network Semantic Embedding (JNNSE) algorithm to combine text corpus statistics with brain activation data.
• Trained the JNNSE model using a large text corpus and neuroimaging data (e.g., fMRI) collected during word reading.
• Evaluated the JNNSE model against established semantic benchmarks and behavioral measures.

Main Results:

• The JNNSE model demonstrated a closer alignment with behavioral measures of semantics compared to traditional VSMs.
• The model successfully predicted semantic representations for words not present in the training corpus.
• The predictive accuracy of the JNNSE model generalized across different brain imaging technologies and individual subjects.

Conclusions:

• The JNNSE algorithm provides a more faithful and comprehensive representation of mental vocabularies by integrating textual and neural data.
• Combining corpus-based and brain-based semantic information offers synergistic benefits for understanding word meaning.
• This neuro-computational approach opens new avenues for research in cognitive science and artificial intelligence.