Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Graph Mining

Background:

• Model pre-training on large text corpora is effective for Natural Language Processing (NLP) tasks.
• Pre-training graph models on large graphs shows promise for downstream graph applications.
• Limited research exists on pre-training models on large heterogeneous graphs with rich textual data.

Purpose of the Study:

• To propose a novel framework for graph-aware language model pre-training (GaLM) on large graph corpora.
• To investigate the fine-tuning of pre-trained models on diverse downstream applications with varying graph schemas.
• To address the gap in pre-training models that integrate both text and graph information from heterogeneous sources.

Main Methods:

• Developed a graph-aware language model pre-training (GaLM) framework.
• Integrated large language models (LLMs) with graph neural networks (GNNs).
• Employed various fine-tuning strategies for downstream tasks on different graph schemas.

Main Results:

• Demonstrated the effectiveness of GaLM through extensive experiments on real-world and public datasets.
• Empirical results show significant benefits of the proposed pre-training approach.
• In-depth analysis provided valuable insights and lessons learned from the experiments.

Conclusions:

• The proposed graph-aware language model pre-training (GaLM) framework is effective for heterogeneous graph data.
• GaLM enhances performance on various downstream graph applications.
• The study offers a new direction for pre-training models in graph mining and NLP.