Expert evaluation of LLM world models: A high-T_c superconductivity case study

Area of Science:

• Materials Science
• Condensed Matter Physics
• Computational Science

Background:

• Large Language Models (LLMs) offer potential for scientific literature review.
• Assessing LLM accuracy and depth in specialized scientific domains is crucial.
• High-temperature cuprates serve as a complex test case for LLM literature comprehension.

Purpose of the Study:

• To evaluate the ability of LLM systems to understand scientific literature at an expert level.
• To compare the performance of various LLM systems, including retrieval-augmented generation (RAG), in answering complex domain-specific questions.
• To develop a robust methodology for assessing LLM performance in scientific reasoning.

Main Methods:

• An expert-curated database of 1,726 papers on high-temperature cuprates was created.
• A set of 67 expert-formulated questions probing deep literature understanding was developed.
• Six LLM systems, including closed models and a custom RAG system, were evaluated using a rubric assessing accuracy, comprehensiveness, and evidential support.

Main Results:

• Two retrieval-augmented generation (RAG) systems, utilizing curated literature, outperformed closed LLM models.
• RAG systems demonstrated superior performance in providing comprehensive and well-supported answers.
• All evaluated LLM systems exhibited both promising capabilities and critical shortcomings in scientific literature understanding.

Conclusions:

• Retrieval-augmented generation (RAG) approaches show significant potential for enhancing LLM performance in specialized scientific literature exploration.
• The developed question set and evaluation rubric provide a valuable tool for assessing expert-level LLM reasoning.
• Further research is needed to address the limitations of current LLM systems in scientific accuracy and comprehensiveness.