Large language models management of complex medication regimens: a case-based evaluation.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Pharmacotherapy Research

Background:

• Large language models (LLMs) show promise in diagnosing complex medical cases.
• Limited research exists on LLM performance in creating evidence-based treatment plans.
• Evaluating LLMs for developing safe and effective intensive care unit (ICU) treatment plans is crucial.

Purpose of the Study:

• To assess the ability of four LLMs to generate safe and efficacious treatment plans for complex ICU patients.
• To evaluate LLM-generated medication regimens for safety and efficacy.

Main Methods:

• Eight high-fidelity ICU patient cases focusing on medication management were developed.
• Four LLMs (ChatGPT-3.5, ChatGPT-4, Claude-2, Llama-2-70b) generated medication regimens.
• Critical care clinicians reviewed regimens for medication errors and treatment appropriateness.

Main Results:

• LLM-generated regimens contained a median of 4.1-6.9 medication errors.
• Life-threatening medication errors ranged from 16.3% to 57.1% across LLMs.
• GPT-4 showed the highest medication continuation rate (67.3%) and fewest life-threatening errors (p < 0.001).

Conclusions:

• Caution is advised when using current LLMs for medication regimens due to identified errors.
• LLMs show potential as clinical decision support tools for complex medication management.
• Domain-specific prompting and rigorous testing are necessary for effective LLM implementation.