Large language models in materials science: assessing RAG evaluation frameworks through graphene synthesis.

Area of Science:

• Materials Science
• Graphene Synthesis

Background:

• Evaluating Retrieval-Augmented Generation (RAG) systems in specialized scientific domains is challenging due to technical complexity and precision needs.
• Existing automated evaluation frameworks may not adequately capture RAG performance in scientific contexts.

Purpose of the Study:

• To systematically analyze and compare automated evaluation frameworks for scientific RAG systems.
• To establish methodological guidelines for evaluating AI in specialized research domains.

Main Methods:

• A comprehensive evaluation protocol was developed using graphene synthesis as a case study.
• Four assessment approaches were compared: RAGAS, BERTScore, LLM-as-a-Judge, and expert human evaluation.
• Performance was assessed across 20 domain-specific questions.

Main Results:

• BERTScore lacked interpretability and score sensitivity for scientific RAG evaluation.
• LLM-as-a-Judge failed to capture the benefits of retrieval augmentation.
• RAGAS successfully identified relative performance improvements from retrieval augmentation, especially in smaller models, but has limitations in absolute score interpretation for scientific content.

Conclusions:

• RAGAS demonstrates potential for evaluating scientific RAG systems, offering insights into retrieval augmentation benefits.
• Automated frameworks require further development to accurately assess AI performance in specialized scientific domains.
• Findings provide critical considerations for researchers deploying AI in science.