Semantic architectures for domain-specific programming AI agents: lessons from a JavaScript semantic assistant.

Area of Science:

• Computer Science
• Artificial Intelligence
• Software Engineering

Background:

• Large language models (LLMs) struggle with domain-specific code comprehension.
• JavaScript debugging and code generation require deep understanding of syntax and semantics.
• Existing methods lack robust domain knowledge integration for LLMs.

Purpose of the Study:

• To develop a semantic ontology-based dataset for fine-tuning LLMs for JavaScript.
• To improve JavaScript debugging and domain-specific code generation capabilities of LLMs.
• To evaluate the effectiveness of ontology-augmented LLMs against baseline and state-of-the-art models.

Main Methods:

• Constructed a semantic ontology detailing JavaScript syntax and logical relationships.
• Created a Q&A dataset from a formal linked database for LLM training.
• Employed parameter-efficient fine-tuning (e.g., LoRA) on a base LLM (LLaMA-3B).
• Evaluated models on code correctness, logical consistency, adaptability, and error detection.

Main Results:

• The ontology-augmented LLM demonstrated superior performance across all evaluation metrics compared to baseline models.
• Significant improvements were observed in domain-specific code comprehension and generation.
• Comparative analysis showed advantages over generic LLMs and retrieval-based techniques.
• Benchmarking against CodeLlama confirmed the enhanced capabilities of the ontology-enhanced approach.

Conclusions:

• Integrating structured semantic knowledge via ontologies significantly enhances LLM performance in programming tasks.
• This approach offers a repeatable method for creating specialized AI systems for programming.
• Publicly accessible resources promote reproducibility and further research in domain-specific AI.