SciCUEval:一个全面的数据集,用于评估大语言模型中的科学背景理解
Jing Yu1,2, Yuqi Tang1,3, Kehua Feng1,4
1ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, China.
Scientific data
|February 26, 2026
概括
一个新的基准,SciCUEval,评估大型语言模型 (LLMs) 在生物和物理等多个领域的科学理解. 它揭示了LLM在科学数据理解方面的优缺点.
科学领域:
- 科学领域包括生物学,化学,物理,生物医学和材料科学.
背景情况:
- 大型语言模型 (LLM) 展示了强大的一般上下文理解.
- 现有的基准不充分评估LLM在专业科学领域的表现.
- 科学数据的复杂性和多样化的模式带来了独特的评估挑战.
研究的目的:
- 引入SciCUEval,这是一个新的基准数据集,用于评估LLM的科学背景理解.
- 提供跨多个科学学科和数据类型的综合评估.
- 确定当前LLM在科学推理中的特定优点和局限性.
主要方法:
- 构建SciCUEval,一个包括十个域特定子数据集的基准.
- 整合各种数据模式:结构化表,知识图和非结构化文本.
- 系统评估四个核心能力:相关信息的识别,信息缺失的检测,多源集成和上下文意识的推断.
主要成果:
- 在SciCUEval.上对最先进的LLMs进行了广泛的评估.
- 细粒度分析显示,在科学领域和任务中,LLM的表现各不相同.
- 确定了LLM在科学背景理解中脱而出和扎的特定领域.
结论:
- 在科学应用中,SciCUEval是推动LLM能力发展的关键工具.
- 该基准为开发更强大,更准确的科学法学士提供了有价值的见解.
- 未来的LLM发展应该集中于提高科学数据解释和推理能力.
相关概念视频
Language and Cognition
881
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
881
Language Development
1.0K
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
1.0K
Components of Language
866
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
866
Language
971
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
971
Improving Translational Accuracy
15.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
15.3K
Improving Translational Accuracy
3.7K
3.7K

