Fine-tuning or prompting on LLMs: evaluating knowledge graph construction task.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Knowledge Representation

Background:

• Knowledge Graph (KG) construction from unstructured text is crucial for data organization and retrieval.
• Large Language Models (LLMs) offer new possibilities for automated KG creation.
• Evaluating different LLM-based KG construction methods is essential for practical application.

Purpose of the Study:

• To compare Zero-Shot Prompting, Few-Shot Prompting, and Fine-Tuning (FT) for Text-to-Knowledge Graph (T2KG) construction.
• To assess the performance of various LLMs, including Llama2, Mistral, and Starling, in T2KG tasks.
• To identify key factors influencing T2KG construction effectiveness and propose future research directions.

Main Methods:

• Experimentation with three prompting strategies: Zero-Shot, Few-Shot, and Fine-Tuning (FT).
• Utilizing state-of-the-art LLMs such as Llama2, Mistral, and Starling for T2KG construction.
• Developing and applying nuanced evaluation metrics for assessing KG quality.

Main Results:

• Fine-Tuning (FT) demonstrated superior performance compared to Zero-Shot and Few-Shot Prompting.
• The size of the training dataset significantly impacts the quality of the constructed knowledge graphs.
• Nuanced evaluation metrics revealed specific strengths and weaknesses of different methods.

Conclusions:

• Fine-Tuning (FT) is a highly effective method for Text-to-Knowledge Graph (T2KG) construction using LLMs.
• Dataset scale is a critical determinant of success in LLM-based KG generation.
• Future work should focus on refining evaluation metrics, incorporating synonym awareness, and exploring LLM-driven data augmentation.