Source framing triggers systematic bias in large language models.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Social Science

Background:

• Large language models (LLMs) are increasingly utilized for text evaluation.
• Concerns exist regarding the consistency, bias, and robustness of LLM judgments.
• The impact of framing effects on LLM evaluations requires thorough investigation.

Purpose of the Study:

• To assess inter- and intramodel agreement among state-of-the-art LLMs in text evaluation.
• To investigate the influence of source attribution (LLM vs. human, nationality) on LLM judgments.
• To identify potential biases in LLM evaluations stemming from framing effects.

Main Methods:

• Evaluated 4800 narrative statements across 24 diverse topics using four advanced LLMs.
• Conducted a total of 192,000 individual assessments.
• Manipulated statement source attribution to human authors of specified nationalities and other LLMs.

Main Results:

• High inter- and intramodel agreement was observed across LLMs for general topic evaluations.
• Source attribution significantly disrupted LLM agreement, demonstrating susceptibility to framing effects.
• Attributing statements to Chinese individuals systematically reduced agreement scores, particularly for the DeepSeek Reasoner model.

Conclusions:

• LLMs exhibit systematic bias when evaluating text influenced by source framing.
• The neutrality and fairness of LLM-mediated information systems are potentially compromised by these framing effects.
• Further research is crucial to mitigate biases and ensure reliable LLM-based text evaluation.