Artificial intelligence for drug information: Evaluating its use in real-world pharmacy practice

Artificial intelligence (AI) is a rapidly growing field within healthcare and is quickly becoming a staple resource for pharmacists for drug information (DI). The primary aim of this study is to determine the accuracy of 2 common AI tools in answering DI questions compared to the accuracy of information provided by University of Michigan Health (UMH) DI pharmacists.

A sample of 300 questions answered by the UMH DI service from July 2022 to June 2023 were included in this evaluation. Each DI query was condensed into a single question for input into each of 2 AI chatbot services: OpenAI's ChatGPT and Google's Gemini. Each question was entered as a new conversation in the AI tools, and initial responses were recorded along with any follow-up responses with references. After all questions were answered, the accuracy of each response and quality of references were adjudicated on a 3-point scale. ChatGPT and Gemini were both completely accurate, with reliable references provided 19% of the time. Fifteen percent of answers by ChatGPT were completely inaccurate and included fake/unreliable references, while 5% of Gemini's answers were categorized as such. The remaining questions (66% for ChatGPT and 76% for Gemini) contained partially correct responses and/or incomplete resources.

This analysis underscores that generative AI tools should not replace healthcare professionals' own research or consultation with DI teams at their institution, as AI tools lack the ability to create evidence-based responses that DI pharmacists can offer. While chatbots may be helpful for certain tasks, they should not be used as a replacement for the work of clinical pharmacists.