对人类的常识知识和大型语言模型进行大规模评估
Tuan Dung Nguyen1, Duncan J Watts1,2,3, Mark E Whiting1,2,4
1Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA.
PNAS nexus
|March 2, 2026
概括
大型语言模型 (LLM) 与常识知识扎,表现低于人类中位数. 在这个AI评估中,较小的开放重量模型显示出与较大的模型相比令人惊的竞争力.
科学领域:
- 人工智能的人工智能
- 认知科学 认知科学
- 自然语言处理自然语言处理.
背景情况:
- 常识知识对人工智能至关重要,但通常使用同质的人类标签进行评估.
- 现有的基准标准假定人类的常识是统一的,但经验证据显示了显著的差异.
- 这种同质性假设未能捕捉到人类理解的多样性.
研究的目的:
- 提出一种用于评估人工智能的常识知识的新方法,特别是大型语言模型 (LLM).
- 将人类的异质性纳入常识判断中的AI评估框架.
- 测量LLM判断和多样化的人类群体的理解之间的对应性.
主要方法:
- 作为个人调查受访者,LLM被评估与人类常识基准相比.
- 此外,LLM还被用作假设人群的模拟器,以评估与真实人群的一致性.
- 评估框架将常识知识与其文化基础联系起来.
主要成果:
- 大多数LLM在被评估为个人受访者时的表现低于人类中位数.
- 在模拟集体协议时,LLM与人类人口的相关性仅为适度.
- 与较大,专有型号相比,较小的,开放式重量型号显示出具有竞争力的性能.
结论:
- 当前的LLM评估可能过度简化了常识知识的复杂性.
- 人工智能模型需要考虑人类知识的异质性和文化基础.
- 将人工智能适应于多样化的人类群体对于强大的常识推理至关重要.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
1.7K
06:48Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
Published on: June 25, 2019
9.9K
相关概念视频
Language and Cognition
891
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
891
Language Development
1.0K
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
1.0K
Natural and Artificial Concepts
622
In psychology, concepts can be divided into two categories: natural and artificial. Natural concepts are formed through direct or indirect experiences. For example, consider the concept of snow. If you live in a place with regular snowfall, such as Essex Junction, Vermont, you know snow through direct experiences. You’ve seen it fall, touched it, shoveled it, and played in it. You recognize its texture, appearance, and even its smell. In contrast, if you live on an island like Saint...
622
Components of Language
870
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
870
Higher Mental Functions of the Brain: Language
4.0K
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
4.0K
Language
972
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
972
