Detecting implicit biases of large language models with Bayesian hypothesis testing.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Social Science

Background:

• Large language models (LLMs) exhibit impressive capabilities but often perpetuate societal biases from training data.
• Detecting and quantifying these implicit biases in LLMs is crucial for responsible AI development.

Purpose of the Study:

• To introduce a novel framework for detecting social bias in LLMs by framing it as a hypothesis testing problem.
• To compare the efficacy of classical statistical tests with Bayesian inference for bias quantification.

Main Methods:

• Reformulated bias detection as a hypothesis testing problem with the null hypothesis representing the absence of implicit bias.
• Utilized binary-choice questions to measure social bias in various LLMs (e.g., ChatGPT, DeepSeek-V3, Llama-3.1-70B).
• Integrated exact binomial tests with Bayesian inference using Bayes factors for bias detection and quantification.

Main Results:

• Bayes factors demonstrate superior ability in quantifying evidence for competing hypotheses compared to the exact binomial test.
• Bayes factors are robust to small sample sizes, offering more reliable bias quantification.
• LLM bias behavior showed consistency across English and French versions of the CrowS-Pairs dataset, with minor variations attributed to socio-cultural contexts.

Conclusions:

• The proposed hypothesis testing framework, particularly with Bayes factors, provides a robust method for detecting and quantifying social biases in LLMs.
• Bayesian inference offers advantages over classical tests in distinguishing evidence of bias from evidence of no bias.
• Cross-lingual consistency in bias suggests underlying patterns, though cultural nuances warrant further investigation.