The simulation of judgment in LLMs

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Information Science

Background:

• Large Language Models (LLMs) are increasingly used in evaluative tasks like credibility assessment.
• Understanding LLM evaluation strategies is crucial due to their growing integration into information systems.
• Divergences between LLM and human evaluation mechanisms require investigation.

Purpose of the Study:

• To benchmark Large Language Models (LLMs) against expert ratings and human judgments in evaluative tasks.
• To analyze the underlying mechanisms and assumptions guiding LLM evaluations.
• To compare LLM evaluation strategies with human reasoning processes.

Main Methods:

• A structured agentic framework was implemented for direct comparison.
• Six LLMs were benchmarked against NewsGuard and Media Bias/Fact Check expert ratings.
• Nonexpert human participants followed the same evaluation procedure as LLMs (criteria selection, content retrieval, justification).

Main Results:

• Despite output alignment, LLMs exhibited consistent differences in observable evaluation criteria.
• LLM evaluations appear influenced by lexical associations and statistical priors, differing from human contextual reasoning.
• A tendency to confuse linguistic form with epistemic reliability was observed, termed 'epistemia'.

Conclusions:

• LLM evaluation strategies may differ significantly from human normative reasoning, leaning towards pattern-based approximation.
• Delegating judgment to LLMs could alter fundamental heuristics in evaluative processes.
• The study raises critical questions about the role and impact of LLMs in complex evaluative functions.