专家对LLM世界模型的评估:一个高Tc超导体实例研究
Haoyu Guo1, Maria Tikhanovskaya2,3, Paul Raccuglia2
1Department of Physics, Cornell University, Ithaca, NY 14853.
概括
大型语言模型 (LLM) 显示了科学文献探索的前景. 使用精选文献的检索增强生成 (RAG) 系统在回答高温杯子的复杂问题方面表现优异,与封闭模型相比.
科学领域:
- 材料科学 材料科学 材料科学
- 凝聚物质物理学 凝聚物质物理学
- 计算科学 计算科学
背景情况:
- 大型语言模型 (LLM) 为科学文献审查提供了潜力.
- 评估LLM在专业科学领域的准确性和深度至关重要.
- 高温 cuprates 作为一个复杂的测试案例LLM文献理解.
研究的目的:
- 评估LLM系统在专家水平上理解科学文献的能力.
- 为了比较各种LLM系统的性能,包括检索增强生成 (RAG),在回答复杂的特定领域问题.
- 开发一个强大的方法来评估科学推理中的LLM绩效.
主要方法:
- 创建了一个由专家策划的数据库,包含1726篇关于高温酸盐的论文.
- 开发了一套由专家制定的67个问题,探讨了深入的文学理解.
- 六个LLM系统,包括封闭模型和定制RAG系统,使用评估准确性,全面性和证据支持的标题进行了评估.
主要成果:
- 两个采用精选文献的检索增强生成 (RAG) 系统的表现优于封闭的LLM模型.
- RAG系统在提供全面和支持良好的答案方面表现出卓越的表现.
- 所有评估的LLM系统都表现出有前途的能力和科学文献理解的关键缺陷.
结论:
- 检索增强生成 (RAG) 方法显示出在专业科学文献探索中提高LLM绩效的巨大潜力.
- 开发的问卷和评估标题为评估专家级LLM推理提供了有价值的工具.
- 需要进一步的研究来解决目前的LLM系统在科学准确性和全面性方面的局限性.
相关概念视频
Types Of Superconductors
1.7K
A superconductor is a substance that offers zero resistance to the electric current when it drops below a critical temperature. Zero resistance is not the only interesting phenomenon as materials reach their transition temperatures. A second effect is the exclusion of magnetic fields. This is known as the Meissner effect. A light, permanent magnet placed over a superconducting sample will levitate in a stable position above the superconductor. High-speed trains that levitate on strong...
1.7K
Superconductor
1.9K
A substance that reaches superconductivity, a state in which magnetic fields cannot penetrate, and there is no electrical resistance, is referred to as a superconductor. In 1911, Heike Kamerlingh Onnes of Leiden University, a Dutch physicist, observed a relation between the temperature and the resistance of the element mercury. The mercury sample was then cooled in liquid helium to study the linear dependence of resistance on temperature. It was observed that, as the temperature decreased, the...
1.9K
Theory of Metallic Conduction
1.9K
The conduction of free electrons inside a conductor is best described by quantum mechanics. However, a classical model makes predictions close to the results of quantum mechanics. It is called the theory of metallic conduction.
In this theory, Newton's second law of motion is used to determine the acceleration of an electron in the presence of an applied electric field. Then, its velocity is expressed via this acceleration.
An electron moves through the crystal, containing positive ions,...
In this theory, Newton's second law of motion is used to determine the acceleration of an electron in the presence of an applied electric field. Then, its velocity is expressed via this acceleration.
An electron moves through the crystal, containing positive ions,...
1.9K
Trends in Lattice Energy: Ion Size and Charge
27.0K
An ionic compound is stable because of the electrostatic attraction between its positive and negative ions. The lattice energy of a compound is a measure of the strength of this attraction. The lattice energy (ΔHlattice) of an ionic compound is defined as the energy required to separate one mole of the solid into its component gaseous ions. For the ionic solid sodium chloride, the lattice energy is the enthalpy change of the process:
27.0K


