Arch-Eval benchmark for assessing chinese architectural domain knowledge in large language models.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Construction Technology

Background:

• Large Language Models (LLMs) are increasingly used in Natural Language Processing (NLP).
• There is a lack of evaluation studies for LLMs specifically within the construction and architectural domains.
• Assessing domain-specific knowledge processing in LLMs is crucial for their practical application.

Purpose of the Study:

• To introduce the "Arch-Eval" framework for evaluating LLMs in architecture.
• To assess the performance of 14 different LLMs on architectural knowledge.
• To analyze LLM response stability and accuracy in the architectural domain.

Main Methods:

• Development of the "Arch-Eval" framework.
• Creation of a standardized dataset with at least 875 questions.
• Testing 14 LLMs over seven iterations, measuring Stability and Accuracy.
• Comparison of Chain-of-Thought (COT) and Answer-Only (AO) evaluation methods.

Main Results:

• Significant performance differences were observed among the 14 evaluated LLMs.
• The average accuracy difference between COT and AO evaluations was less than 3%.
• COT evaluation response time was substantially longer (26x) than AO (62.23s vs. 2.38s per question).

Conclusions:

• The "Arch-Eval" framework provides a reliable method for assessing LLMs in architecture.
• LLM performance in architectural knowledge question-answering varies significantly.
• Future research should focus on domain customization, reasoning, and multimodal interaction for LLMs in construction.