Development and evaluation of an agentic LLM based RAG framework for evidence-based patient education

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Health Informatics

Background:

• Developing high-quality, evidence-based patient education materials (PEMs) is crucial for effective healthcare communication.
• Large language models (LLMs) offer potential for automating content generation, but require careful evaluation for accuracy and safety, especially in specialized languages like Arabic.

Purpose of the Study:

• To develop and evaluate an agentic retrieval augmented generation (ARAG) framework for generating Arabic PEMs using open-source LLMs.
• To assess LLMs' capability to act as validation agents for blocking harmful content within generated PEMs.

Main Methods:

• Twelve LLMs were tested across four setups: base, base with prompt engineering, ARAG, and ARAG with prompt engineering.
• PEM quality was evaluated using automated LLM assessment and expert review across five metrics: accuracy, readability, comprehensiveness, appropriateness, and safety.
• Validation agent performance was assessed using a dataset of harmful/safe PEMs to measure blocking accuracy.

Main Results:

• ARAG setups enhanced generation performance for 10 out of 12 LLMs, with Arabic-focused models dominating the top ranks.
• AceGPT-v2-32B with ARAG and prompt engineering demonstrated the highest performance.
• Validation agent accuracy strongly correlated with model size (≥27B parameters needed for >0.80 accuracy), though Fanar-7B excelled in generation but not validation.

Conclusions:

• Arabic-centric LLMs show advantages for Arabic PEM generation.
• The ARAG framework significantly improves PEM generation quality, though context limitations affect large-context models.
• Model size is a critical factor for reliable harmful content validation by LLMs.