LEAF: Learning and Evaluation Augmented by Fact-Checking to Improve Factualness in Large Language Models.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) exhibit limitations in factual accuracy, particularly within knowledge-intensive fields such as healthcare.
• Ensuring factual correctness is critical for reliable medical question answering systems.

Purpose of the Study:

• To introduce LEAF (Learning and Evaluation Augmented by Fact-Checking), a novel framework designed to enhance the factuality of LLMs in medical question answering.
• To address the challenge of LLM factual inaccuracies in specialized domains.

Main Methods:

• LEAF integrates three core components: RAFE (Robust Accurate Fact-checking Engine) for response evaluation, Fact-Check-then-RAG for retrieval guidance, and Learning from Fact Check for self-training.
• RAFE utilizes open-source LLMs and domain-specific retrieval for accuracy assessment.
• Fact-Check-then-RAG employs fact-checking outcomes to refine retrieval processes without altering model parameters.

Main Results:

• RAFE demonstrated superior performance in identifying inaccuracies compared to existing methods like Factcheck-GPT.
• The Fact-Check-then-RAG approach effectively corrected response errors.
• The Learning from Fact Check component improved LLM performance through self-training, even without requiring labeled data.

Conclusions:

• LEAF significantly enhances LLM factuality in medical question answering, offering a scalable solution for industrial applications demanding high accuracy.
• Real-world deployment in healthcare demonstrated an 83% improvement in factuality scores, validating LEAF's practical applicability for adapting LLMs to specific organizational knowledge bases.