Bringing Light Into the Dark: A Large-Scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Heterogeneity in knowledge graph embedding (KGE) model implementations hinders fair comparisons.
• Lack of standardized evaluation makes assessing KGE model reproducibility difficult.

Purpose of the Study:

• To assess the reproducibility of 21 published KGE models.
• To identify factors influencing KGE model performance and establish best practices.
• To provide a comprehensive benchmark for KGE research.

Main Methods:

• Re-implementation and evaluation of 21 KGE models using the PyKEEN software package.
• Systematic benchmarking across four datasets, involving thousands of experiments and significant computational resources (24,804 GPU hours).
• Analysis of reproducibility based on reported and adjusted hyperparameters.

Main Results:

• Identified which KGE model results were reproducible with original hyperparameters, required adjustments, or were irreproducible.
• Demonstrated that model architecture, training approach, loss function, and inverse relation modeling collectively impact performance.
• Found that several KGE architectures can achieve state-of-the-art results with careful configuration.

Conclusions:

• Careful configuration, including architecture, training, loss function, and inverse relations, is critical for KGE model performance.
• Reproducibility in KGE research can be improved through standardized practices and open-source tools.
• Best practices and configurations for KGE models are provided, enabling more reliable and competitive research.