Enhancing Ontology Engineering with Large Language Models: A Stage-Wise Human-in-the-Loop Study

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Knowledge Representation and Reasoning

Background:

• Manual ontology engineering is labor-intensive and requires specialized expertise.
• Ensuring semantic interoperability in digital healthcare relies on structured knowledge models.
• Large language models (LLMs) offer potential for automating and enhancing knowledge modelling tasks.

Purpose of the Study:

• To evaluate the effectiveness of LLMs in assisting ontology development within a human-in-the-loop framework.
• To investigate LLM performance across different stages of ontology engineering: term extraction, class definition, and relation specification.
• To compare LLM-assisted methods against manual ontology construction.

Main Methods:

• Integrating LLMs at three key checkpoints: term extraction, class/hierarchy definition, and relation specification.
• Testing two experimental settings: LLM-only generation and LLM with manual input.
• Evaluating outputs using precision, recall, and F1 metrics against a manually created ontology.

Main Results:

• Human-in-the-loop input consistently enhanced the quality of LLM-generated ontology components.
• LLMs demonstrated stronger performance in the initial conceptualization phases (term extraction, class definition) compared to formalization stages.
• The study established a reproducible framework for evaluating LLM-assisted ontology engineering.

Conclusions:

• LLMs and human validation offer complementary strengths for efficient and accurate ontology development.
• Generative AI can be practically integrated into ontology-driven health data modeling and standardization workflows.
• This research provides guidance for leveraging AI in biomedical knowledge engineering.