Evaluating a generative artificial intelligence accuracy in providing medication instructions from smartphone images

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Patient Education Technology

Background:

• The Food and Drug Administration (FDA) requires patient labeling like Medication Guides (MG) and Instructions for Use (IFU) for safe medication practices.
• Low health literacy and complex material navigation can lead to medication errors, therapy failure, and adverse events.
• Generative AI offers a novel approach for scalable, personalized patient education using image recognition and text generation.

Purpose of the Study:

• To assess the accuracy and safety of ChatGPT-generated medication instructions derived from user-submitted drug images.
• To compare AI-generated instructions against official manufacturer Instructions for Use (IFU) and Medication Guides (MG).

Main Methods:

• Images of 12 multi-step administration medications were input into ChatGPT's image recognition feature.
• Text classifiers, Count Vectorization (CountVec), and Term Frequency-Inverse Document Frequency (TF-IDF) were used to compare AI responses with official documentation.
• Independent pharmacists evaluated the clinical accuracy and validity of ChatGPT's generated instructions for patient use.

Main Results:

• ChatGPT accurately identified all medications and produced corresponding instructions.
• CountVec demonstrated superior text similarity (76% average) compared to TF-IDF.
• Clinical evaluation highlighted significant deficiencies in AI-generated instructions for complex medications, impacting accuracy.

Conclusions:

• ChatGPT demonstrates promise for generating patient-friendly medication instructions, but performance is contingent on medication complexity.
• Further development and rigorous clinical validation are essential to ensure the safety and reliability of AI-driven medical guidance, especially for intricate drug administration protocols.