SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models.

Area of Science:

• Scientific domains including biology, chemistry, physics, biomedicine, and materials science.

Background:

• Large Language Models (LLMs) demonstrate strong general contextual understanding.
• Existing benchmarks inadequately assess LLM performance in specialized scientific domains.
• Scientific data complexity and diverse modalities pose unique evaluation challenges.

Purpose of the Study:

• To introduce SciCUEval, a novel benchmark dataset for evaluating LLMs' scientific context understanding.
• To provide a comprehensive assessment across multiple scientific disciplines and data types.
• To identify specific strengths and limitations of current LLMs in scientific reasoning.

Main Methods:

• Construction of SciCUEval, a benchmark comprising ten domain-specific sub-datasets.
• Integration of diverse data modalities: structured tables, knowledge graphs, and unstructured texts.
• Systematic evaluation of four core competencies: relevant information identification, information-absence detection, multi-source integration, and context-aware inference.

Main Results:

• Extensive evaluations of state-of-the-art LLMs were conducted on SciCUEval.
• Fine-grained analysis revealed varying LLM performance across scientific domains and tasks.
• Identified specific areas where LLMs excel and struggle in scientific context understanding.

Conclusions:

• SciCUEval serves as a crucial tool for advancing LLM capabilities in scientific applications.
• The benchmark provides valuable insights for developing more robust and accurate scientific LLMs.
• Future LLM development should focus on enhancing scientific data interpretation and reasoning abilities.