Evaluating Large Language Models and Retrieval-Augmented Generation Enhancement for Delivering Guideline-Adherent...

Area of Science:

• Artificial Intelligence in Healthcare
• Nutritional Science
• Cardiovascular Disease Prevention

Background:

• Cardiovascular disease (CVD) is a leading global cause of mortality.
• Accessible and reliable online information on cardiovascular (CV) health and nutrition is limited.
• Large Language Models (LLMs) offer potential for scalable health information but require validation for accuracy, especially regarding nutrition for CVD prevention.

Purpose of the Study:

• To assess the effectiveness of standard (off-the-shelf) and retrieval-augmented generation (RAG)-enhanced LLMs in providing guideline-adherent nutrition information for CVD prevention.
• Compared three off-the-shelf LLMs (ChatGPT-4o, Perplexity, Llama 3-70B) against a RAG-enhanced Llama 3-70B model.

Main Methods:

• Developed a knowledge bank based on the American Heart Association's 2021 dietary guidelines.
• Used RAG to enhance the Llama 3-70B model with the knowledge bank and specific prompts.
• Evaluated 30 expert-approved nutrition questions posed to each LLM using criteria like appropriateness, reliability, readability, harm, and guideline adherence.
• Statistical analysis included ANOVA for mean scores and Cohen κ for interrater agreement.

Main Results:

• The Llama 3 model integrated with RAG significantly outperformed off-the-shelf models (GPT-4o, Perplexity, Llama 3) in reliability, appropriateness, and guideline adherence.
• The RAG-enhanced model demonstrated no harmful responses, unlike some off-the-shelf models.
• High interrater agreement (κ>70%) confirmed the robustness of the expert evaluations.

Conclusions:

• RAG integration substantially enhances LLM performance for delivering evidence-based nutrition information for CVD prevention.
• Off-the-shelf LLMs exhibit limitations and potential risks in providing accurate dietary guidance.
• Future applications of RAG in LLMs can improve the accessibility and reliability of crucial health information.