Hallucination, monofacts, and miscalibration: An empirical investigation

Hallucinated facts in large language models have recently been shown to obey a statistical lower bound determined by the monofact rate (related to the classical Good-Turing missing mass estimator) minus model miscalibration [A. T. Kalai, S. S. Vempala, "Calibrated language models must hallucinate" in Proceedings of the 56th Annual ACM Symposium on Theory of Computing (STOC) (New York, NY, USA, 2024), pp. 160-171]. We present empirical investigation of this three-way relationship in classical [Formula: see text]-gram models and fine-tuned transformer models. By generating training data from Pareto distributions with varying shape parameters, we systematically control the monofact rate and establish its positive relationship with hallucination. To bridge theory and practice, we derive an empirical analog of the hallucination bound by replacing the population miscalibration term (Section 1.1) with an empirical bin-wise Kullback-Leibler (KL) divergence and confirm its practical viability. We then introduce selective upweighting-a simple yet effective technique that strategically repeats as little as 5% of training examples-to deliberately inject miscalibration into the model. This intervention reduces hallucination by up to 40%, challenging universal deduplication policies. Our experiments reveal a critical trade-off: selective upweighting maintains preinjection levels of accuracy while substantially reducing hallucination, whereas standard training gradually improves accuracy but fails to address persistently high hallucination, indicating an inherent tension in optimization objectives.