Evaluating Large Language Model-Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Clinical Communication

Background:

• E-prescribing systems aim to improve medication safety and efficiency.
• Current electronic prescription (e-prescription) instructions often lack clarity and personalization.
• Large language models (LLMs) offer potential for enhancing medication information delivery.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs) in generating improved medication instructions for e-prescriptions.
• To assess the impact of different prompt strategies on the clarity, personalization, and bias of LLM-generated instructions.
• To enhance patient understanding and adherence through better medication communication.

Main Methods:

• Developed patient-centered guidelines for medication instructions.
• Created a dataset of 104 outpatient scenarios based on Brazilian e-prescribing standards.
• Utilized closed-source and open-source LLMs with varying prompt strategies (generic, enhanced, bias-mitigated).
• Assessed LLM outputs using automated metrics and human evaluation.

Main Results:

• The bias-mitigated prompt significantly improved instruction adequacy and acceptability (94.3%).
• Personalization was rated highly, particularly with enhanced and bias-mitigated prompts.
• Factual errors and hallucinations were infrequent; bias was mitigated by specific prompts and LLM choice.
• Both LLMs demonstrated comparable adequacy, with varying hallucination frequencies.

Conclusions:

• LLM-supported generation shows promise for creating effective prescription directions.
• Optimized prompts can enhance clarity, personalization, and reduce bias in e-prescription instructions.
• LLMs can improve communication between healthcare providers and patients within e-prescribing systems.