Prompt Engineering of Large Language Models for Medication Dose Calculation

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Pharmacology

Background:

• Electronic Health Records (EHRs) contain vast amounts of medication data.
• Manual extraction of medication dosing from EHRs is time-consuming and prone to errors.
• Automated methods are needed to efficiently and accurately capture dosing information.

Purpose of the Study:

• To design and validate a large language model (LLM) framework for systematic medication dose extraction from EHR data.
• To evaluate the performance of five different LLMs in extracting medication dosing information across multiple therapeutic classes.
• To assess the impact of prompt engineering on the accuracy of LLM-based dose extraction.

Main Methods:

• Manual dose annotations were performed on 4295 medications across nine therapeutic classes.
• Five LLMs (Mistral-small, Llama 3-70B, Nova lite, DeepSeek, Claude 3.5 Sonnet) were tested using iterative prompt engineering.
• Performance was evaluated using R² and classification accuracy, comparing model-derived doses against manual annotations.

Main Results:

• Claude 3.5 Sonnet and DeepSeek demonstrated the highest performance, with Claude achieving R² of 99.32% and accuracy of 92.36% on training data.
• Prompt optimization led to minor improvements in Claude and DeepSeek, and significant gains in Nova lite, Llama, and Mistral-small.
• In testing data, Claude and DeepSeek achieved R² of 99.74% and 95.58% respectively, indicating high accuracy in automated medication dose extraction.

Conclusions:

• LLMs show high accuracy and potential for automated medication dose extraction from EHR data.
• The developed LLM framework can augment clinical practice through improved medication reconciliation.
• This study provides a foundation for leveraging LLMs to enhance clinical research and data analysis.