Soft-Community Kernel Rényi Spectrum for Semantic Uncertainty Estimation in Large Language Models.

Area of Science:

• Artificial Intelligence
• Information Theory
• Natural Language Processing

Background:

• Uncertainty estimation is crucial for safe deployment of large language models (LLMs).
• Current methods use hard clustering and von Neumann entropy, which are sensitive to noise and clustering order.
• These limitations hinder reliable uncertainty quantification in critical applications.

Purpose of the Study:

• To develop a principled information-theoretic framework for LLM semantic uncertainty estimation.
• To address the limitations of existing entropy-based methods.
• To provide more stable and discriminative uncertainty estimates.

Main Methods:

• Constructing a weighted semantic graph from pairwise similarity scores of LLM generations.
• Inferring soft community assignments using weighted graph community detection.
• Quantifying uncertainty via Rényi entropy of the kernel spectrum derived from soft assignments.

Main Results:

• The proposed Rényi spectral uncertainty framework demonstrates improved robustness to semantic noise.
• The method shows reduced dependence on clustering heuristics and greater flexibility via its order parameter.
• Experiments on question answering tasks confirm more stable and discriminative uncertainty estimates, especially with limited sampling.

Conclusions:

• The novel framework offers a principled and flexible approach to LLM uncertainty estimation.
• Rényi spectral uncertainty provides a tunable measure interpolating between dominant and diverse semantic modes.
• This advancement is particularly valuable for safety-sensitive and decision-critical LLM applications.