Use of Retrieval-Augmented Large Language Model for COVID-19 Fact-Checking: Development and Usability Study

Area of Science:

• Artificial Intelligence
• Public Health Informatics
• Computational Linguistics

Background:

• The COVID-19 pandemic fueled an "infodemic" of misinformation, overwhelming traditional fact-checking.
• Large language models (LLMs) offer scalable solutions but are prone to generating hallucinations.
• Limitations in LLM reliability hinder effective, large-scale misinformation combat.

Purpose of the Study:

• To enhance COVID-19 fact-checking accuracy and reliability.
• To address LLM hallucination and context inaccuracy using retrieval-augmented generation (RAG).
• To evaluate RAG-enhanced LLM performance against misinformation.

Main Methods:

• Developed RAG-enhanced models (naïve RAG, LOTR-RAG, CRAG, SRAG) integrated with GPT-4.
• Utilized a dataset of ~130,000 COVID-19 peer-reviewed papers for context.
• Evaluated models on real-world and synthesized datasets (500 claims each) for accuracy, F1-score, precision, and sensitivity.

Main Results:

• RAG models significantly improved accuracy over baseline GPT-4 on both datasets.
• CRAG and SRAG models achieved the highest accuracies (0.972 and 0.973 on real-world; 0.978 on synthesized).
• RAG consistently reduced hallucinations and improved contextual accuracy, especially CRAG and SRAG.

Conclusions:

• Integrating RAG systems with LLMs substantially boosts automated fact-checking accuracy and relevance.
• This approach offers rapid, reliable information verification to combat public health misinformation.
• RAG enhances transparency by citing sources, crucial for trust during health crises.