PharmacyGPT: exploration of artificial intelligence for medication management in the intensive care unit

The purpose of this evaluation was to develop a novel framework for large language model (LLMs) iterative prompt optimization for the specialized domain of medication decision-making in the intensive care unit (ICU). This serves as a first step towards the use of LLMs as clinical decision support (CDS) tools capable of performing pharmacy related tasks.

Using a cohort of 1,000 adult patients managed in the ICU for greater than 24 h, an iterative optimization process in the GPT-4 framework was applied to enhance performance of various medication-related tasks, including patient disease state clustering by medication regimen, medication regimen generation, and outcome prediction with the intent to develop PharmacyGPT. Within this feedback loop, the input prompts were adjusted based on model performance in previous iterations.

The iterative prompt optimization process housed within PharmacyGPT was able to develop meaningful disease state clusters based on input information, develop medication regimens with inclusion of drug, dose, and frequency, and predict mortality with an accuracy of 0.75, precision 0.37, and recall 0.70.

Iterative prompt optimization shows promise as a rapid means to improve LLM functionality to specific tasks, even in highly domain specific areas like medication management in the ICU. Such domain specific engineering may serve as a strategy to develop LLMs as viable CDS tools.