Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) exhibit input-dependent inconsistencies, highlighting the need for robust uncertainty quantification.
• Existing methods often focus on single models, neglecting the potential benefits of leveraging model diversity for improved reliability.

Purpose of the Study:

• To propose and evaluate MUSE (Multi-LLM Uncertainty via Subset Ensembles), a novel approach for quantifying LLM uncertainty by exploiting model diversity.
• To demonstrate that aggregating outputs from diverse LLMs yields more reliable uncertainty estimates compared to individual models.

Main Methods:

• MUSE employs an information-theoretic approach, specifically Jensen-Shannon Divergence, to identify and aggregate well-calibrated subsets of LLMs.
• The method leverages hypothesized complementary predictions from LLMs due to variations in training data and the Zipfian nature of language.

Main Results:

• Experiments on binary prediction tasks show that MUSE significantly improves calibration and predictive performance over single-model and basic ensemble approaches.
• The study validates the hypothesis that diverse LLM outputs, when aggregated effectively, lead to superior uncertainty quantification.

Conclusions:

• MUSE offers an effective strategy for enhancing the reliability and calibration of large language models, particularly in high-stakes scenarios.
• The proposed method provides a foundation for further research into ensemble techniques for uncertainty quantification in AI systems.