Evaluating large language models for pharmacotherapy simulations: a mixed-methods study

Area of Science:

• Pharmacy Education
• Artificial Intelligence in Medicine
• Clinical Simulation

Background:

• Simulation-based learning is crucial for clinical pharmacy education but faces scalability challenges due to faculty resource limitations.
• Large language models (LLMs) present a potential solution for scalable simulation generation, but their pedagogical effectiveness and clinical reliability need thorough evaluation.

Purpose of the Study:

• To evaluate the pedagogical rigor and clinical reliability of LLM-generated clinical pharmacy simulations.
• To compare student satisfaction with LLM-based simulations versus traditional methods.
• To identify specific areas of weakness in LLM-generated clinical case simulations.

Main Methods:

• A mixed-methods, counterbalanced study involving 104 PharmD students interacting with LLM-generated acute myeloid leukemia (AML) or chronic myeloid leukemia (CML) cases.
• Expert panel evaluation of simulation sessions based on clinical authenticity, instructional design, and clinical reasoning.
• Student satisfaction surveys comparing LLM-based simulations to traditional methods.

Main Results:

• Over half (51.5%) of the 103 evaluated LLM-generated sessions met passing criteria across all domains.
• Clinical accuracy and safety were the most significant limiting factors (58.3%), while clinical reasoning (81.6%) and instructional design (82.5%) scored higher.
• Chronic myeloid leukemia (CML) simulations performed significantly better (62.3%) than acute myeloid leukemia (AML) simulations (40.0%).
• Identified errors included guideline misalignment, pharmacotherapeutic inaccuracies, fabricated evidence, and cross-condition recommendations, predominantly in AML cases.
• Students reported higher satisfaction with LLM-based simulations (49.8%) compared to traditional methods (30.0%), but this did not correlate with expert-assessed quality.

Conclusions:

• LLM-generated simulations show promise for clinical pharmacy education but require robust expert oversight and validation for clinical accuracy and safety.
• Platform-specific and disease-specific validation are essential before widespread educational deployment.
• Further effectiveness trials are needed to assess objective learning outcomes.