GraKerformer: A Transformer With Graph Kernel for Unsupervised Graph Representation Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Graph Representation Learning

Background:

• Transformer models excel in NLP but struggle with unsupervised graph representation learning (UGRL).
• Existing UGRL methods often focus on local graph structures, limiting their ability to capture global information and leading to poor generalization.
• A need exists for advanced models that can effectively represent complex graph structures.

Purpose of the Study:

• To introduce GraKerformer, a novel Transformer-based model designed to improve unsupervised graph representation learning.
• To address the limitations of conventional UGRL approaches in capturing comprehensive graph structural information.
• To enhance the performance and generalization capabilities of models in UGRL tasks.

Main Methods:

• Proposed GraKerformer, a modified Transformer architecture incorporating graph neural networks.
• Utilized the shortest-path graph kernel (SPGK) to weight attention scores within the Transformer.
• Integrated SPGK with graph neural networks to encode nuanced graph structural information.

Main Results:

• GraKerformer demonstrated superior performance in unsupervised graph representation learning tasks.
• The model effectively captured comprehensive structural information of graphs, overcoming limitations of local-structure focused methods.
• Evaluations on benchmark graph classification datasets validated the enhanced performance.

Conclusions:

• GraKerformer offers a significant advancement in unsupervised graph representation learning.
• The integration of SPGK and GNNs within a Transformer framework effectively encodes complex graph structures.
• The proposed model achieves state-of-the-art performance, paving the way for more robust graph-based deep learning applications.