Rx-LLM: a benchmarking suite to evaluate safe large language model performance for medication-related tasks.

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics
• Pharmacology

Background:

• Large language models (LLMs) require specialized benchmarks for safe medication management.
• Comprehensive Medication Management (CMM) is the standard for medication optimization.
• Current LLMs need evaluation for clinical medication tasks.

Purpose of the Study:

• To design CMM-aligned benchmarks for LLM evaluation.
• To quantify baseline performance of leading LLMs on medication-related tasks.

Main Methods:

• Developed six CMM-process benchmarks: formulation, order generation, route, drug-drug interaction, renal dose, and indication matching.
• Created 250 clinician-annotated data pairs per benchmark.
• Evaluated GPT-4o-mini, MedGemma-27B, and LLaMA3-70B using zero-shot prompting and task-specific metrics.

Main Results:

• LLaMA3-70B led in formulation, order generation, route, and indication matching.
• GPT-4o-mini achieved highest accuracy in drug-drug interaction identification and renal dose adjustment.
• Consistency varied widely across models and tasks.

Conclusions:

• LLM performance differs significantly across medication-related tasks.
• LLaMA3-70B shows promise for formulation, ordering, route, and indication tasks.
• GPT-4o-mini demonstrates potential for drug interaction and renal dose adjustments.
• Task-specific evaluation is crucial for clinical decision support LLM deployment.