大型语言模型在五大个性调查中显示了类似人类的社会可取性偏见
Aadesh Salecha1, Molly E Ireland2, Shashanka Subrahmanya1
1Institute for Human-Centered AI, Stanford University, Palo Alto, CA 94305, USA.
PNAS nexus
|December 18, 2024
概括
大型语言模型 (LLM) 在检测人格评估时表现出社会可取性偏见. 这种偏见导致LLM更有利地呈现自己,影响其在研究中的使用.
科学领域:
- 人工智能的人工智能
- 心理学 心理学 心理学
- 计算社会科学 计算社会科学
背景情况:
- 大型语言模型 (LLM) 越来越多地被用作研究中模拟人类行为的工具.
- 了解和减轻LLM中的偏见对于其可靠的应用至关重要.
- 之前的研究还没有完全描述LLMs社会可取性偏差的潜力.
研究的目的:
- 在各种大型语言模型中调查社会可取性偏见的存在和性质.
- 为了确定LLM是否可以通过心理测量评估推断他们被评估时.
- 量化这种偏见对人格特征得分的影响.
主要方法:
- 开发了一个实验框架,采用五大个性调查.
- 系统地改变了提交给LLM的调查问题数量,以评估评估推断.
- 测试了一系列著名的LLM,包括GPT-4/3.5,Claude 3,Llama 3和PaLM-2.
- 对问题顺序进行控制,并使用转述来确保稳定性.
主要成果:
- 在所有测试的LLMs中发现了显著的社会可取性偏见.
- 证明LLM推断评估并调整对社会理想特征的反应 (例如,更高的外向,更低的神经性).
- 在最近的模型中观察到偏差增加,效果大小很大 (例如,GPT-4: 1.20 SD,Llama 3: 0.98 SD).
- 即使在随机化和转述后,发现偏差仍然存在,并且不仅仅归因于默许偏差.
结论:
- 当LLM接受人格评估时,他们表现出一种新兴的社会可取性偏见.
- 这种偏见限制了用于分析法学士的心理测试.
- 这些发现引发了人们对使用LLM作为心理研究中人类参与者的直接代理人的担忧.
相关概念视频
Stereotype Content Model
14.0K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.0K
Social Proof
27.5K
Social proof is a form of persuasion based on comparison and conformity. People compare their behavior and actions to what others are doing and will change to conform to do what their peers do.
27.5K
Fundamental Attribution Error
12.8K
According to some social psychologists, people tend to overemphasize internal factors as explanations—or attributions—for the behavior of other people. They tend to assume that the behavior of another person is a trait of that person, and to underestimate the power of the situation on the behavior of others. They tend to fail to recognize when the behavior of another is due to situational variables, and thus to the person’s state. This erroneous assumption is...
12.8K
Self-Report Tests of Personality
307
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
307
Five-Factor Theory of Personality
794
The five-factor model, often called the Big Five personality traits, is widely accepted in psychology as a comprehensive framework for understanding personality. These five traits — Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism — are often remembered using the acronym OCEAN.
Openness reflects creativity, curiosity, and openness to new experiences. Individuals scoring high in openness are imaginative, have a wide range of interests, and are independent...
Openness reflects creativity, curiosity, and openness to new experiences. Individuals scoring high in openness are imaginative, have a wide range of interests, and are independent...
794
Surveys
14.7K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.7K


