When to Pre-Train Graph Neural Networks? From Data Generation Perspective!

Area of Science:

• Graph representation learning
• Machine learning theory

Background:

• Graph pre-training aims to transfer knowledge from unlabeled data to improve downstream tasks.
• Negative transfer remains a significant challenge, limiting the effectiveness of current graph pre-training methods.
• Existing research focuses on 'what' and 'how' to pre-train, but overlooks 'when' pre-training is advantageous.

Purpose of the Study:

• To introduce a generic framework, W2PGNN, that determines the optimal situations for applying graph pre-training.
• To address the critical question of 'when to pre-train' before engaging in computationally intensive pre-training or fine-tuning.
• To explore the generative mechanisms connecting pre-training and downstream data.

Main Methods:

• W2PGNN fits pre-training data into graphon bases, identifying fundamental transferable patterns.
• It defines a generator space using convex combinations of graphon bases.
• The feasibility of pre-training is quantified by the generation probability of downstream data from this space.

Main Results:

• W2PGNN provides a theoretically sound method for defining the application scope of graph pre-trained models.
• Empirical evidence supports its ability to quantify pre-training feasibility and guide pre-training data selection.
• The framework helps identify scenarios where graph pre-training yields distinct benefits, mitigating negative transfer.

Conclusions:

• W2PGNN offers a novel approach to assess the utility of graph pre-training, moving beyond 'what' and 'how' to 'when'.
• The framework enables informed decisions on applying graph pre-trained models, enhancing downstream task performance.
• It provides a principled way to avoid negative transfer and maximize the benefits of graph pre-training.