Analyzing Llama 3-based Approach for Axiom Translation from Ontologies

Area of Science:

• Computer Science
• Artificial Intelligence
• Knowledge Representation

Background:

• Ontology development relies on collaboration between engineers and domain experts.
• Translating complex ontology axioms into natural language improves understandability for non-expert stakeholders.
• Evaluating ontologies is crucial for ensuring their accuracy and utility.

Purpose of the Study:

• To investigate the efficacy of large language models (LLMs) in translating ontology axioms into natural language.
• To assess the potential of LLM-generated translations to facilitate ontology evaluation.
• To identify the strengths and limitations of LLMs in this axiom translation task.

Main Methods:

• Utilized Llama 3, a large language model, for axiom translation.
• Translated 1,192 ontology axioms across 19 distinct types from five published ontologies.
• Conducted a manual evaluation of the translated axioms for accuracy and representational quality.

Main Results:

• 13.67% of translated axioms were fully accurate, while 22.48% were inaccurate.
• A significant portion, 63.84%, of translations were partially accurate.
• LLMs showed competence in generating hierarchical natural language equivalents but struggled with complex axioms.

Conclusions:

• LLMs show promise for supporting knowledge engineering in ontologies through axiom translation.
• Opportunities exist for enhancing LLM performance via few-shot training or integration into knowledge engineering workflows.
• Further research is needed to refine LLM capabilities for accurate and comprehensive ontology axiom translation.