Explicitly unbiased large language models still form biased associations.

Area of Science:

• Artificial Intelligence
• Psychology
• Sociology

Background:

• Large language models (LLMs) can exhibit implicit social biases, similar to humans, even when passing explicit bias tests.
• Measuring implicit bias in LLMs is challenging due to proprietary models and the need to assess impact on decisions.

Purpose of the Study:

• To introduce novel prompt-based measures for detecting implicit biases in LLMs.
• To assess the presence and nature of implicit stereotype biases in value-aligned LLMs.

Main Methods:

• Developed the LLM Word Association Test, adapting the Implicit Association Test for prompt-based bias detection.
• Introduced the LLM Relative Decision Test to evaluate subtle discrimination in contextual AI decisions.
• Applied these methods to 8 value-aligned LLMs across race, gender, religion, and health categories.

Main Results:

• Pervasive stereotype biases were found in the tested LLMs, reflecting societal biases.
• Biases were identified in 21 specific stereotypes, including associations with race, gender, age, and health.
• The prompt-based measures successfully exposed nuanced biases in proprietary LLMs that standard benchmarks missed.

Conclusions:

• Novel prompt-based psychological measures can effectively reveal implicit biases in LLMs.
• LLMs, even those aligned with values, harbor societal biases that impact their decisions.
• These findings highlight the need for advanced methods to ensure AI fairness and mitigate harmful stereotypes.