Comparison of Drug Information Questions' Responses Between Drug Information Pharmacists and Artificial Intelligence...

Drug Information Centers (DICs) play a critical role in health care by providing accurate, evidence-based responses to drug-related inquiries. The integration of artificial intelligence (AI) into this domain offers opportunities to enhance efficiency and decision-making. This study aims to compare the accuracy and safety of drug information responses provided by drug information pharmacists and artificial intelligence (AI) technologies, including GPT-4o, Gemini, Copilot, and Medical Chat (OpenAI, San Francisco, CA, USA; Google, Mountain View, CA, USA; Microsoft, Redmond, WA, USA; Medical Chat, developer information not publicly specified).

A retrospective, real-world study was conducted over 3 months at a tertiary hospital. The end points of this study are the statistical differences in overall accuracy of responses obtained from generative AI technologies and drug information pharmacists to drug information questions' responses measured in six dimensions.

GPT-4o and Medical Chat achieved composite scores comparable to DIC pharmacists (mean [standard deviation (SD)]: 18.3 [2.5], 17.5 [3.1], and 17.7 [3.5], respectively), while Gemini and Copilot showed lower performance (16.7 [3.5] and 16.4 [2.9]). Welch's analysis of variance (ANOVA) demonstrated a significant difference in composite scores among groups (p < 0.001), with GPT-4o outperforming Gemini and Copilot in pairwise comparisons. Differences were also observed across question categories, with routine inquiries such as administration and dosing being more common than complex categories. Inter-rater agreement was low (Fleiss' kappa = 0.015, p = 0.297), indicating variability in scoring among evaluators.

Some AI tools demonstrated performance comparable with pharmacists for routine drug information questions, while others showed lower reliability, particularly for clinically nuanced responses. Variability among tools and low rater agreement highlight the need for cautious interpretation and continued pharmacist oversight when using AI in drug information practice.