Expert of Experts Verification and Alignment (EVAL) Framework for Large Language Models Safety in Gastroenterology.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Clinical Decision Support

Background:

• Large language models (LLMs) offer potential for medical question answering but carry risks due to inaccurate outputs.
• Manual grading of LLM responses is impractical for clinical settings, hindering safe implementation.
• Ensuring LLM accuracy is crucial for high-stakes medical decision-making.

Purpose of the Study:

• To introduce EVAL (Expert-of-Experts Verification and Alignment), a system designed to streamline LLM output evaluation.
• To enhance the safety and accuracy of LLMs in medical applications, specifically for upper gastrointestinal bleeding (UGIB).
• To compare various LLM configurations and evaluation techniques.

Main Methods:

• Evaluated multiple LLMs (GPT, Claude, LLaMA, Mixtral) across 27 configurations, including zero-shot, retrieval-augmented generation, and supervised fine-tuning.
• Implemented EVAL using similarity-based ranking with Fine-Tuned ColBERT and a reward model trained on human-graded responses for rejection sampling.
• Assessed alignment with human performance using three separate datasets.

Main Results:

• Fine-Tuned ColBERT demonstrated the highest alignment with human performance (ρ = 0.81-0.91).
• The reward model accurately replicated human grading in 87.9% of cases.
• Rejection sampling using EVAL significantly improved LLM accuracy by 8.36% overall.

Conclusions:

• EVAL provides a scalable solution for assessing LLM accuracy in medical contexts.
• The developed system enhances LLM safety and reliability for clinical decision support.
• This approach is vital for the responsible deployment of AI in healthcare.