Topology only pre-training: towards generalised multi-domain graph models.

Area of Science:

• Graph representation learning
• Machine learning
• Artificial intelligence

Background:

• Unsupervised representation learning allows model fine-tuning with limited data.
• Current graph representation learning methods are domain-specific, hindering cross-domain transfer.
• This limitation restricts the application of pre-trained graph models to new domains.

Purpose of the Study:

• To introduce Topology Only Pre-Training (ToP), a novel graph pre-training method.
• To demonstrate ToP's effectiveness in enabling cross-domain transfer learning.
• To challenge existing assumptions about feature requirements in graph pre-training.

Main Methods:

• Developed ToP, a graph pre-training technique that excludes node and edge features.
• Evaluated ToP on diverse datasets from multiple domains, including out-of-domain examples.
• Compared ToP's performance against supervised baselines and other generalist graph models.

Main Results:

• ToP models achieved significantly better performance than supervised baselines in 75% of experiments.
• Positive transfer learning was observed in 85.7% of tasks when using node and edge features during fine-tuning.
• Out-of-domain pre-training topologies proved more beneficial than in-domain ones for certain tasks.
• ToP demonstrated superior transfer learning on molecular benchmarks compared to molecule-specific pre-training.
• ToP performed strongly against other generalist graph models, irrespective of model size.

Conclusions:

• ToP facilitates effective transfer learning in graph representation learning, particularly in data-scarce scenarios.
• The method demonstrates broad applicability across various domains, including those not seen during pre-training.
• ToP's success opens new research avenues in graph foundation models and cross-domain transfer learning.