Uncertainty in GNN Learning Evaluations: A Comparison between Measures for Quantifying Randomness in GNN Community...

Area of Science:

• Graph Neural Networks (GNNs)
• Machine Learning
• Network Analysis

Background:

• Unsupervised community detection using GNNs leverages both graph connectivity and feature information.
• Accurate evaluation of GNN performance for community detection is complicated by numerous influencing factors.
• Identifying latent communities has broad applications in social networks, genomics, and beyond.

Purpose of the Study:

• To evaluate the impact of hyperparameter optimization on GNN performance in unsupervised community detection.
• To compare the consistency and quality of algorithm rankings across different evaluation metrics.
• To identify the most robust metric for assessing randomness in GNN performance evaluations.

Main Methods:

• Comparison of GNN performance using hyperparameter optimization versus default hyperparameters.
• Evaluation of three distinct metrics for assessing the consistency of algorithm rankings under randomness.
• Assessment of the Wasserstein distance (W randomness coefficient) for quantifying randomness.

Main Results:

• Neglecting hyperparameter optimization leads to a significant loss in GNN performance.
• Ties in algorithm ranks can substantially affect the quantification of randomness.
• The Wasserstein distance provides the most robust assessment of randomness compared to other metrics.

Conclusions:

• Standardized evaluation criteria are essential for reproducible GNN performance reporting.
• Hyperparameter tuning is critical for achieving optimal GNN performance in community detection tasks.
• The W randomness coefficient offers a reliable method for evaluating the stability of GNN-based community detection algorithms.