Related Experiment Video
Updated: Sep 20, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Comparison of a generative large language model to pharmacy student performance on therapeutics examinations
Christopher J Edwards1, Bernadette Cornelison1, Brian L Erstad1
1Department of Pharmacy Practice & Science, University of Arizona R. Ken Coit College of Pharmacy, Tucson, AZ, United States of America.
Objective:
To compare the performance of a generative language model (ChatGPT-3.5) to pharmacy students on therapeutics examinations.
Methods:
Questions were drawn from two pharmacotherapeutics courses in a 4-year PharmD program. Questions were classified as case based or non-case based and application or recall. Questions were entered into ChatGPT version 3.5 and responses were scored. ChatGPT's score for each exam was calculated by dividing the number of correct responses by the total number of questions. The mean composite score for ChatGPT was calculated by adding individual scores from each exam and dividing by the number of exams. The mean composite score for the students was calculated by dividing the sum of the mean class performance on each exam divided by the number of exams. Chi-square was used to identify factors associated with incorrect responses from ChatGPT.
Results:
The mean composite score across 6 exams for ChatGPT was 53 (SD = 19.2) compared to 82 (SD = 4) for the pharmacy students (p = 0.0048). ChatGPT answered 51 % of questions correctly. ChatGPT was less likely to answer application-based questions correctly compared to recall-based questions (44 % vs 80 %) and less likely to answer case-based questions correctly compared to non-case-based questions (45 % vs 74 %).
Conclusion:
ChatGPT scored lower than the average grade for pharmacy students and was less likely to answer application-based and case-based questions correctly. These findings provide valuable insight into how this technology will perform which can help to inform best practices for item development and helps highlight the limitations of this technology.
More Related Videos
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
10:17Improving Student Outcomes with an Adaptable Molecular Cloning Course-Based Undergraduate Research Experience
Published on: November 15, 2024
Related Concept Videos
Pharmacokinetic Models: Overview
There are three primary types of models: empirical, compartment, and physiological. Empirical models, with minimal...
Pharmacokinetic Models: Comparison and Selection Criterion
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
Drug Administration and Therapy Phases: Overview
The pharmaceutical phase focuses on leveraging the physicochemical properties of the drug to design and manufacture an effective product. Variants include orally administered tablets or capsules, topical creams or ointments, and parenteral-delivery solutions or emulsions.
The pharmacokinetic phase...
Three-Compartment Open Model
Analysis of Population Pharmacokinetic Data
Bioequivalence: Overview