Refining LLMs outputs with iterative consensus ensemble (ICE).

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Medical Informatics

Background:

• Large language models (LLMs) demonstrate potential in complex tasks like medical question answering.
• However, LLM performance gains can plateau, and reliability remains a concern.
• Existing methods may require specialized models or complex fusion techniques.

Purpose of the Study:

• To introduce and evaluate the Iterative Consensus Ensemble (ICE) framework for improving LLM accuracy and reliability.
• To assess ICE's effectiveness across diverse datasets, including medical and PhD-level reasoning tasks.
• To demonstrate a cost-efficient approach to enhancing LLM performance without specialized components.

Main Methods:

• Developed the Iterative Consensus Ensemble (ICE) framework, utilizing iterative reasoning and feedback among multiple LLMs.
• Tested ICE on over 4000 multiple-choice questions from primary care exams, medical benchmarks, and a PhD-level reasoning dataset.
• Employed standard LLMs and repeated prompting, avoiding specialized reward models or token-level fusion.

Main Results:

• ICE improved overall accuracy by up to 27% compared to single-model attempts.
• Achieved 81% accuracy on medical subsets and 72% on multi-domain tasks.
• Significantly boosted performance on the GPQA-diamond benchmark from 46.9% to 68.2% (over 45% relative gain).

Conclusions:

• Iterative collaboration among LLMs via ICE enhances reliability and accuracy in reasoning tasks.
• ICE offers a cost-effective alternative to complex models, achieving comparable results on specialized datasets.
• The framework shows promise for advancing LLM capabilities in medical and general reasoning domains.