SynEL: A synthetic benchmark for entity linking.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Knowledge Representation

Background:

• Large language models (LLMs) show promise for building knowledge graphs from text across various domains.
• LLM performance is hampered by domain-specific language, necessitating specialized benchmarks for knowledge graph construction tasks.
• Existing domain-specific benchmarks for evaluating LLMs in knowledge graph construction are limited.

Purpose of the Study:

• To introduce SynEL, a novel benchmark for evaluating text-based knowledge extraction methods.
• To address the scarcity of domain-specific benchmarks for LLM evaluation.
• To propose and validate methods for generating synthetic datasets to improve LLM performance.

Main Methods:

• Developed a comprehensive methodology for constructing the SynEL benchmark.
• Proposed two distinct approaches for generating synthetic datasets.
• Evaluated LLM performance on knowledge extraction tasks using the SynEL benchmark and synthetic data.
• Assessed accumulated hallucinations in LLM outputs.

Main Results:

• Existing LLMs exhibit a significant performance decrease (up to 25 absolute points in micro-F1 scores) when extracting low-resource entities compared to high-resource entities.
• Incorporating synthetic datasets into the training process led to an improvement of up to 10 absolute points in micro-F1 scores.
• The SynEL benchmark was validated using customer support dialogues.

Conclusions:

• Specialized benchmarks like SynEL are crucial for accurately assessing and optimizing LLMs for domain-specific knowledge graph construction.
• Synthetic data generation offers a viable strategy to enhance LLM performance in low-resource scenarios.
• The public release of SynEL and its generation code facilitates further research and development in LLM-based knowledge extraction.