WalkLM: A Uniform Language Model Fine-tuning Framework for Attributed Graph Embedding.

Area of Science:

• Graph representation learning
• Machine learning
• Artificial intelligence

Background:

• Real-world graphs possess complex attributes and structures challenging for traditional methods.
• Graph neural networks (GNNs) often require extensive training for specific downstream tasks.
• Existing unsupervised methods struggle with uniform modeling of diverse graph attributes.

Purpose of the Study:

• To develop a novel, unsupervised framework for generic graph representation learning.
• To simultaneously model complex attributes and flexible structures of real-world graphs.
• To obtain powerful graph embeddings not limited to specific downstream predictions.

Main Methods:

• Integration of language models (LMs) and random walks (RWs).
• Performing attributed random walks (RWs) on graphs.
• Automated composition of textual sequences from RWs for LM fine-tuning.
• Extraction of node embeddings from fine-tuned LMs capturing attribute semantics and graph structures.

Main Results:

• The proposed framework achieves significant improvements in downstream prediction tasks.
• Learned node embeddings demonstrate superior performance across multiple real-world attributed graph datasets.
• Outperforms a comprehensive set of state-of-the-art unsupervised node embedding methods.

Conclusions:

• The LM and RW integrated framework offers a powerful, data-efficient approach to graph representation learning.
• This method provides generic, unsupervised graph representations adaptable to various tasks.
• Opens new avenues for leveraging LMs in complex real-world graph modeling.