Development and Content Analysis Protocol for Evaluating Artificial Intelligence in Drug-Related Information

Artificial intelligence (AI) has significant transformative potential across various sectors, particularly in health care. This study aims to develop a protocol for the content analysis of a method designed to assess AI applications in drug-related information, specifically focusing on contraindications, adverse reactions, and drug interactions. By addressing existing challenges, this preliminary research seeks to enhance the safe and reliable integration of AI into healthcare practices.

A study protocol was developed for the creation of the method, followed by an initial content analysis conducted by an expert panel. The method was established in phases: (1) Analysis of drug-related databases and form development; (2) AI configuration; (3) Expert panel review and initial validation.

In Phase 1, the Micromedex, UpToDate, and Medscape databases were reviewed to establish terminology and classifications related to contraindications, adverse reactions, and drug interactions, resulting in the development of a questionnaire for the AI. Phase 2 involved configuring the Gemini AI tool to enhance response specificity. In Phase 3, AI responses to 30 questions were validated by an expert panel, yielding a 76.7% agreement rate for appropriateness, while 23.3% were deemed inappropriate, particularly concerning contraindicated drug interactions.

This preliminary study demonstrates the potential for using an AI-powered tool to standardize drug-related information retrieval, particularly for contraindications and adverse reactions. While AI responses were generally appropriate, improvements are needed in identifying contraindicated drug interactions. Further research with larger datasets and broader evaluations is required to enhance AI's reliability in healthcare settings.