在潜在内容分析中,比较大型语言模型和人类注释者,分析情绪,政治倾向,情绪强度和刺
Ljubiša Bojić1,2, Olga Zagovora3,4, Asta Zelenkauskaite5,6
1Institute for Artificial Intelligence Research and Development of Serbia, Fruskogorska, Novi Sad, Serbia. ljubisa.bojic@ivi.ac.rs.
Scientific reports
|April 3, 2025
概括
大型语言模型 (LLM) 在情绪和政治倾向分析方面显示出高可靠性,通常超过人类注释者. 虽然LLM证明了时间的一致性,但人类的专业知识仍然对微妙的情绪强度解释至关重要.
科学领域:
- 自然语言处理自然语言处理.
- 人工智能的人工智能
- 计算语言学 计算语言学
背景情况:
- 数字文本的指数增长需要自动化方法来进行潜在内容分析.
- 大型语言模型 (LLM) 为内容分析提供了一个有前途的途径,但它们与人类注释者的比较性能需要彻底调查.
研究的目的:
- 评价与人类注释者对最先进的LLM的可靠性,一致性和质量.
- 通过情绪分析,政治倾向,情绪强度和刺检测来比较LLM的表现.
主要方法:
- 七位高级LLM (包括GPT-4,Gemini,Llama-3.1-70B,Mixtral 8×7B) 和33名人类注释者分析了100个文本项目.
- 总共收集了3300个人类注释和19200个LLM注释.
- 在三个时间点评估了LLM的时间一致性.
主要成果:
- 在情绪和政治倾向方面,LLM和人类表现出高的评级者间可靠性,LLM的表现更可靠.
- 在情绪强度分析中,LLM表现出比人类更高的可靠性,尽管人类的强度评级更高.
- 无论是LLM还是人类都在刺言论检测方面扎,显示出低可靠性;LLM显示出出色的时间一致性.
结论:
- 在情绪和政治倾向分析方面,LLM,特别是GPT-4,可以有效地匹配人类的表现.
- 人类的专业知识对于解释情绪强度至关重要.
- 在特定领域,LLM显示出对一致,高质量的潜在内容分析的巨大潜力.
相关概念视频
02:16Stereotype Content Model
13.9K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
13.9K
02:55Stereotypes, Prejudice, and Discrimination
89.7K
Humans are very diverse and although we share many similarities, we also have many differences. The social groups we belong to help form our identities (Tajfel, 1974). These differences may be difficult for some people to reconcile, which may lead to prejudice toward people who are different. Prejudice is a negative attitude and feeling toward an individual based solely on one’s membership in a particular social group (Allport, 1954; Brown, 2010). Prejudice is common against people who...
89.7K
01:54Attitudes
28.0K
Attitude is our evaluation of a person, an idea, or an object. We have attitudes for many things ranging from products that we might pick up in the supermarket to people around the world to political policies. Typically, attitudes are favorable or unfavorable: positive or negative (Eagly & Chaiken, 1993). And, they have three components: an affective component (feelings), a behavioral component (the effect of the attitude on behavior), and a cognitive component (belief and knowledge;...
28.0K
01:28Lateralization
281
Brain lateralization refers to the division of mental processes and functions between the two hemispheres of the brain, a phenomenon that optimizes neural efficiency and underpins complex abilities in humans. This specialization allows each hemisphere to perform tasks where it has a comparative advantage, facilitating more refined cognitive capabilities across different domains.
281
00:56Attribution Theory
12.9K
Behavior is a product of both the situation (e.g., cultural influences, social roles, and the presence of bystanders) and of the person (e.g., personality characteristics). Subfields of psychology tend to focus on one influence or behavior over others. Situationism is the view that our behavior and actions are determined by our immediate environment and surroundings. In contrast, dispositionism holds that our behavior is determined by internal factors (Heider, 1958).
12.9K
02:13The Representativeness Heuristic
15.7K
The representative heuristic describes a biased way of thinking, in which you unintentionally stereotype someone or something. For example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation, because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.
15.7K