Enhanced LLM-supported instructions for medication use through retrieval-augmented generation

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Patient Communication

Background:

• Current prescription systems face challenges in personalizing patient instructions for clarity.
• Large Language Models (LLMs) show potential for improving healthcare communication.

Purpose of the Study:

• To investigate the effectiveness of LLM Retrieval-Augmented Generation (RAG) using patient information leaflets (PIL) for generating clearer medication instructions.
• To evaluate different prompting strategies for LLM-generated patient instructions.

Main Methods:

• Utilized a database of 119 outpatient scenarios for LLM input.
• Evaluated open-source Llama3 with standard, enhanced, and RAG-enhanced prompts incorporating Brazilian regulatory PIL data.
• Assessed generated instructions by five physicians for adequacy, clarity, personalization, and quality; computed cosine similarity.

Main Results:

• RAG significantly improved instruction adequacy (p=0.001) and clarity (p=0.012).
• Enhanced prompts with RAG reduced critical errors like incorrect or incomplete instructions and factual inaccuracies.
• Prompt engineering minimized vague and unsolicited instructions, with minimal hallucination.

Conclusions:

• AI, particularly RAG with authoritative PIL data, can enhance medication safety and patient-friendly communication.
• Human validation remains crucial for ensuring error-free and safe implementation of AI-generated medical information.