Evaluating the Accuracy of Large Language Models in Pharmaceutical Calculations: A Comparison of ChatGPT and MathGPT

Generative Artificial Intelligence is increasingly integrated into education, with large language models such as ChatGPT and MathGPT providing students with quick access to answers for pharmaceutical calculation problems. However, concerns persist regarding their accuracy for critical skills like pharmaceutical calculations. This study evaluates the accuracy of ChatGPT 3.5 and MathGPT Unlimited in solving pharmacy calculation problems relevant to student pharmacist curricula.

Fifty pharmaceutical calculation questions covering 9 North American Pharmacist Licensure Examination-aligned topic categories were selected from the University of Arizona R. Ken Coit College of Pharmacy curriculum. Responses generated by ChatGPT 3.5 (free version) and MathGPT Unlimited (data collection concluded July 2024) were compared against an instructor-validated answer key.

Both ChatGPT and MathGPT correctly answered 35 of the 50 questions, with no significant difference between them. Accuracy varied across topics, with lower performance in sodium chloride Equivalence and Solutions. Logistic regression revealed no significant association between accuracy and question type or topic. Common errors included improper unit conversions, misapplication of calculation methods (eg, creatinine clearance with adjusted body weight), and variations in precision.

While demonstrating some capability, the 70% accuracy of ChatGPT 3.5 and MathGPT Unlimited, coupled with fundamental calculation errors, indicates they are insufficiently accurate for sole use in pharmaceutical calculations. These findings highlight the imperative for pharmacy educators to emphasize foundational skills and teach critical evaluation of artificial intelligence-generated responses.