A multi-layer retrieval-augmented large language model framework for enhancing hypertension education | JoVE Visualize

Area of Science:

Artificial Intelligence in Healthcare
Medical Information Retrieval
Natural Language Processing

Background:

Large Language Models (LLMs) show promise for health information retrieval but suffer from "hallucinations," posing security risks.
Ensuring accuracy and reliability in AI-driven health education is crucial for patient safety and effective information dissemination.
Existing LLMs require domain-specific enhancements to mitigate risks associated with medical information generation.

Purpose of the Study:

To enhance the accuracy and reliability of Large Language Models (LLMs) for hypertension education.
To develop and evaluate the HEART (Hypertension Enhancing Answer Retrieval Tool) framework, integrating Retrieval-Augmented Generation (RAG) technology.
To assess the impact of RAG integration on the performance of leading LLMs in responding to hypertension-related queries.

Main Methods:

A specialized hypertension knowledge base was constructed and integrated into a Retrieval-Augmented Generation (RAG) system, forming the HEART framework.
Four prominent LLMs (ChatGPT-4o, Claude-3.5, Gemini-2.5, Llama-3.3) were evaluated using a set of 50 common hypertension questions, both in their base form and enhanced with HEART.
Clinical experts assessed LLM responses based on accuracy, completeness, consistency, robustness, security, and overall quality.

Main Results:

Integration with the HEART framework significantly improved all four LLMs across accuracy, completeness, consistency, security, and robustness (P < 0.05).
Average overall quality scores increased substantially for all models: Llama-3.3 (3.57 to 4.20), Claude-3.5 (3.92 to 4.38), ChatGPT-4o (3.91 to 4.32), and Gemini-2.5 (4.03 to 4.38) (P < 0.001).
The HEART framework effectively addressed LLM hallucinations, enhancing the trustworthiness of AI-generated health information.

Conclusions:

Combining high-quality, domain-specific medical data with advanced AI techniques like RAG is vital for improving healthcare applications.
The HEART framework demonstrates a successful strategy for mitigating LLM inaccuracies and enhancing reliability in hypertension education.
This approach offers a pathway to more secure and dependable AI-powered health information systems.