在大型语言模型中评估中国建筑领域知识的Arch-Eval基准
Jie Wu1, Mincheng Jiang1, Juntian Fan1
1Key Laboratory of Disaster Prevention and Structural Safety of Ministry of Education, Guangxi Key Laboratory of Disaster Prevention and Engineering Safety, College of Civil Engineering and Architecture, Guangxi University, 530004, Nanning, China.
Scientific reports
|April 18, 2025
概括
本研究引入了Arch-Eval来评估建筑中的大型语言模型 (LLM). 虽然LLM的表现各不相同,但链式思维评估更准确,但比只回答更慢.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 建筑工程技术 建筑工程技术
背景情况:
- 大型语言模型 (LLM) 越来越多地用于自然语言处理 (NLP).
- 缺乏对LLM的评估研究,特别是在建筑和建筑领域.
- 在LLM中评估特定领域的知识处理对于其实际应用至关重要.
研究的目的:
- 引入"Arch-Eval"框架用于评估建筑中的LLMs.
- 评估14个不同的LLM在建筑知识方面的表现.
- 在建筑领域分析LLM响应稳定性和准确性.
主要方法:
- 开发了"Arch-Eval"框架的开发.
- 创建一个标准化的数据集,至少有875个问题.
- 在7次代中测试14个LLM,测量稳定性和准确性.
- 思考链 (COT) 和只有答案 (AO) 评估方法的比较.
主要成果:
- 在14个评估的LLM中观察到显著的绩效差异.
- COT和AO评估之间的平均准确性差异小于3%.
- 对COT评估的响应时间比AO (26倍) 长得多 (62.23s vs 2.38s).
结论:
- "Arch-Eval"框架为评估建筑学LLM提供了一种可靠的方法.
- 在建筑知识问题和答案的LLM表现有很大的不同.
- 未来的研究应该专注于领域定制,推理和建筑LLM的多式联络交互.
相关概念视频
Spanning Openings in Brick Walls
148
In brick wall construction, supporting structures are crucial for openings like windows and doors to maintain the integrity and support the weight of the wall above. These supports include lintels, corbels, and arches, each serving specific structural purposes.
Lintels are primary supports used to span openings and can be crafted from materials such as reinforced concrete, steel-reinforced brick masonry, or simple steel angles. These are straightforward to install and are typically concealed...
Lintels are primary supports used to span openings and can be crafted from materials such as reinforced concrete, steel-reinforced brick masonry, or simple steel angles. These are straightforward to install and are typically concealed...
148
Structural Classification of Joints
3.0K
Joints, also known as articulations, are classified based on their structural characteristics, i.e., based on whether the articulating surfaces of the adjacent bones are directly connected by fibrous connective tissue or cartilage, or whether the articulating surfaces contact each other within a fluid-filled joint cavity. These differences serve to divide the joints of the body into three structural classifications.
A fibrous joint is where the adjacent bones are united by fibrous connective...
A fibrous joint is where the adjacent bones are united by fibrous connective...
3.0K
Design Example: Dimensioning of Concrete Masonry Construction
78
For the construction of a storeroom using concrete masonry units, it's essential to align the dimensions of the structure with the actual sizes of the blocks and the intended mortar joints. On the site in question, there's a stockpile of concrete masonry blocks with a nominal size of eight by eight by sixteen inches, which are to be used in the construction of the storeroom.
The site engineer has laid out a plan for the storeroom with external dimensions of twelve feet in length and...
The site engineer has laid out a plan for the storeroom with external dimensions of twelve feet in length and...
78
Archival Research
15.9K
Some researchers gain access to large amounts of data without interacting with a single research participant. Instead, they use existing records to answer various research questions. This type of research approach is known as archival research. Archival research relies on looking at past records or data sets to look for interesting patterns or relationships. For example, a researcher might access the academic records of all individuals who enrolled in college within the past ten years and...
15.9K
Typical Model Studies
155
Fluid mechanics model studies often utilize scaled-down systems to predict fluid behavior in full-scale environments, such as river flows, dam spillways, and structures interacting with open surfaces. Maintaining Froude number similarity in river models is crucial, as it replicates surface flow features like wave patterns and velocities.
155
Improving Translational Accuracy
8.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.5K


