Related Experiment Video
Updated: Jun 23, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Comparison of Large Language Model-Based Systems and Prompt Engineering for Internal Medicine Clinical Pharmacy Cases
Samuel S Yang1, Clement E Ng1, Hyunuk Seung2
1University of Maryland Medical Center, Baltimore, Maryland, USA.
Summary
Evaluating artificial intelligence in clinical pharmacy, this study found neither domain-specific large language models (LLMs) nor prompt engineering improved response accuracy or completeness. However, one LLM demonstrated superior reference validity for practice applications.
Area of Science:
- Artificial Intelligence in Healthcare
- Clinical Pharmacy Informatics
- Large Language Model (LLM) Applications
Background:
- Large Language Models (LLMs) are AI tools generating responses with variable accuracy and completeness.
- Methods to enhance LLM response quality in clinical pharmacy practice require further evaluation.
Purpose of the Study:
- To evaluate the impact of domain-specific LLMs versus general-purpose LLMs on response quality.
- To assess the utility of prompt engineering templates in refining LLM outputs for clinical pharmacy.
- To compare the accuracy, completeness, and reference validity of different LLM systems.
Main Methods:
- A prospective, observational study utilized 50 clinical case questions.
- Two LLM systems were compared: ChatGPT 4o (general-purpose) and OpenEvidence (healthcare-specific with RAG).
- A two-by-two factorial design assessed the effect of prompt engineering templates on LLM responses, evaluated by pharmacists.
Main Results:
- No statistically significant interaction was found between LLM type and prompt engineering for accuracy or completeness.
- OpenEvidence demonstrated significantly higher reference validity compared to ChatGPT (p < 0.001).
- Predicted probabilities for the primary outcome (accuracy and completeness) varied across conditions, with OpenEvidence without a template showing 0.64.
Conclusions:
- Neither domain-specific LLMs with RAG nor prompt engineering templates significantly improved LLM accuracy or completeness in this clinical pharmacy context.
- OpenEvidence's superior reference validity suggests potential for clinical practice integration.
- Further research is needed to optimize LLM utility as AI adoption grows in healthcare.
Related Concept Videos
01:26Pharmacokinetic Models: Comparison and Selection Criterion
Physiological and compartmental models are valuable tools used in studying biological systems. These models rely on differential equations to maintain mass balance within the system, ensuring an accurate representation of the dynamic processes at play.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
01:17Introduction to Language of Pathophysiology ll
This lesson explores key terms that describe how diseases progress, their outcomes, and their distribution in populations.Diagnostic tests identify diseases and monitor treatment. These include blood and urine tests, biopsies, imaging (X-ray, MRI), and detection of infectious agents.Remission is a reduction or disappearance of symptoms.Exacerbation refers to the worsening of symptoms, such as increased wheezing during an asthma attack.A precipitating factor triggers an acute episode, while a...
01:14Pharmacodynamic Models: Link Model and Systems Pharmacodynamic Model
The link model is a fundamental pharmacokinetic-pharmacodynamic (PK–PD) approach to account for delayed drug responses when the observed effect does not immediately correlate with the drug's plasma concentration peak. This delay is mathematically addressed by introducing an effect compartment concentration, Ce, which is kinetically linked to the plasma concentration, Cp, via a first-order rate constant, ke0. The linkage allows for a more accurate prediction of drug effects over time. A higher...
01:27Pharmacodynamic Models: Overview
Pharmacodynamic (PD) responses describe the interaction between a drug and its biological target, culminating in a physiological effect. These responses can be classified into different types: continuous variables, such as blood glucose levels; categorical outcomes, like survival rates; and time-to-event metrics, such as disease progression. Understanding and modeling PD responses are critical for optimizing drug efficacy and safety.PD models describe the relationship between drug concentration...
01:15Methods of Documentation VI: Case Management Model
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic illness...
For example, a patient with a chronic illness...
01:29Pharmacodynamic Models: Direct Effect Model and Indirect Response Model
Pharmacodynamic models are essential tools in understanding the relationship between drug concentrations and their effects on biological systems. By characterizing the dynamics of drug action, these models guide dose selection, optimize therapeutic efficacy, and inform the development of new drugs. Two major classes of pharmacodynamic models include direct effect and indirect response models.Direct Effect ModelsDirect effect models describe the immediate relationship between drug concentration...