在语言理解任务上测试AI揭示了对底层含义的不敏感性
Vittoria Dentella1,2, Fritz Günther3, Elliot Murphy4
1Universitat Rovira i Virgili, Tarragona, Spain. vittoria.dentella@unipv.it.
Scientific reports
|November 14, 2024
概括
大型语言模型 (LLM) 在语言理解任务上展示了偶然的准确性,显着低于人类的表现. 目前的人工智能模型缺乏类似人类的语言理解,这是由于组合推理的潜在缺陷.
科学领域:
- 人工智能的人工智能
- 计算语言学 计算语言学
- 认知科学 认知科学
背景情况:
- 大型语言模型 (LLM) 越来越多地用于各种应用,导致声称具有类似人类的语言能力.
- 莫拉维克的悖论强调了在人工智能中复制类似人类技能的困难,即使是看似简单的任务.
研究的目的:
- 系统地评估最先进的法学士的语言理解和推理能力.
- 将LLM绩效与人类基线进行比较,使用一种评估语言理解的新基准.
主要方法:
- 七个最先进的LLM在理解问题的基准上进行了测试.
- 在两个设置 (单词和开放长度回复) 中,在26680个数据点的数据集中提示模型.
- 通过在相同的提示下测试400个人来建立人类基线.
主要成果:
- LLM的准确性是随机的,并且在答案中表现出相当大的变化.
- 人类参与者在定量准确性方面显著超过了所有测试的LLM.
- 定性分析揭示了LLM语言理解中的非人类错误.
结论:
- 当前的LLM,尽管有其实用性,但并不具有类似人类的语言理解能力.
- 缺乏语法和语义信息的组成运算符可能解释了LLM的局限性.
- 需要进一步的研究来弥合人工智能和人类语言能力之间的差距.
更多相关视频
08:32Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
Published on: September 5, 2019
5.6K
10:11Portable Intermodal Preferential Looking IPL: Investigating Language Comprehension in Typically Developing Toddlers and Young Children with Autism
Published on: December 14, 2012
18.4K
相关概念视频
Language and Cognition
325
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
325
Higher Mental Functions of the Brain: Language
744
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
744
Intelligence
6.7K
The term "intelligence" is complex because it refers to both behavior and individuals, and its interpretation varies across cultures. European Americans tend to link intelligence with reasoning and cognitive skills, while in Kenya, it is tied to responsible participation in family and social life. In Uganda, intelligence is seen as the ability to know the right actions and carry them out effectively, while the Iatmul people of Papua New Guinea associate it with the capacity to remember...
6.7K
Stereotypes, Prejudice, and Discrimination
90.0K
Humans are very diverse and although we share many similarities, we also have many differences. The social groups we belong to help form our identities (Tajfel, 1974). These differences may be difficult for some people to reconcile, which may lead to prejudice toward people who are different. Prejudice is a negative attitude and feeling toward an individual based solely on one’s membership in a particular social group (Allport, 1954; Brown, 2010). Prejudice is common against people who...
90.0K
Fundamental Attribution Error
12.8K
According to some social psychologists, people tend to overemphasize internal factors as explanations—or attributions—for the behavior of other people. They tend to assume that the behavior of another person is a trait of that person, and to underestimate the power of the situation on the behavior of others. They tend to fail to recognize when the behavior of another is due to situational variables, and thus to the person’s state. This erroneous assumption is...
12.8K
Cognitive Learning
220
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
220
