Evaluating large language models for accuracy incentivizes hallucinations.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) frequently generate confident but incorrect information, known as hallucinations, which hinders their reliability.
• Existing mitigation strategies like retrieval, tool use, self-verification, and reinforcement learning have not fully resolved the hallucination problem in advanced LLMs.

Purpose of the Study:

• To identify how next-word prediction and accuracy-based evaluations inadvertently encourage LLMs to guess.
• To propose novel evaluation methods that incentivize accurate responses and reduce hallucinations.

Main Methods:

• Utilizing learning theory to demonstrate how next-word pretraining creates a statistical pressure towards hallucination, especially for infrequent facts.
• Analyzing how standard accuracy metrics favor guessing over uncertainty, thereby exacerbating the hallucination issue.
• Proposing "open-rubric" evaluations that explicitly define error penalties to test model abstention and accuracy trade-offs.

Main Results:

• Next-word prediction inherently pressures models towards hallucination, particularly for unique data points.
• Accuracy-based evaluations systematically reward guessing, discouraging models from admitting uncertainty.
• Open-rubric evaluations can be used to realign model incentives, promoting reliability.

Conclusions:

• Hallucinations in LLMs can be reframed as an incentive problem stemming from training and evaluation methodologies.
• Implementing open-rubric evaluations offers a practical pathway to developing more reliable LLMs by correcting misaligned incentives.