Syntactically-informed word representations from graph neural network.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Deep language understanding models primarily use word representations based on distributional knowledge from large text corpora.
• These models often overlook syntactic structural information, despite its proven benefits in various NLP tasks.
• Existing methods require training language models from scratch to incorporate syntactic information.

Purpose of the Study:

• To propose novel syntactically-informed word representations (SIWRs) that enrich pre-trained word embeddings with syntactic information.
• To enable the integration of syntactic knowledge without the need for training language models from scratch.
• To evaluate the effectiveness of SIWRs on downstream information extraction tasks.

Main Methods:

• Developed a graph-based neural model to build SIWRs upon existing static (e.g., GloVe) or contextualized (e.g., ELMo, BERT) word representations.
• Pre-trained the model using a modest amount of task-independent data annotated with syntactic information.
• Extracted SIWRs from intermediate layers of the pre-trained model when applied to downstream task data.

Main Results:

• SIWRs demonstrated performance gains over base representations in nested named entity recognition (NER) and binary/n-ary relation extractions (REs).
• Achieved a 3-9% relative error reduction across these NLP tasks.
• SIWRs outperformed fine-tuning BERT for binary relation extraction tasks.

Conclusions:

• SIWRs offer an effective approach to incorporate syntactic information into deep language understanding models.
• The proposed method enhances performance on critical information extraction tasks.
• SIWRs provide a valuable alternative to traditional methods for enriching word representations with syntactic structure.