Will artificial intelligence chatbots replace clinical pharmacologists? An exploratory study in clinical practice

Area of Science:

• Pharmacovigilance and Drug Safety
• Artificial Intelligence in Medicine
• Natural Language Processing Applications

Background:

• Growing interest in utilizing advanced AI like ChatGPT for medical applications.
• Need to assess the reliability of AI tools for specialized medical information.
• Pharmacovigilance centers handle critical drug safety inquiries.

Purpose of the Study:

• To evaluate the performance of OpenAI's chatbot (GPT 4.0) for drug information tasks.
• Assessing the chatbot's utility within a pharmacovigilance center setting.
• Comparing AI-generated responses with expert pharmacovigilance assessments.

Main Methods:

• Utilized a set of 50 randomly selected drug information questions.
• Questions were sourced from healthcare professionals and patients.
• Compared GPT 4.0 responses against those from pharmacovigilance specialists.

Main Results:

• Chatbot responses were generally deemed unacceptable for pharmacovigilance use.
• Inquiries concerning drug causality assessment yielded imprecise and clinically insignificant answers.
• The AI's performance did not meet the required standards for medical drug information.

Conclusions:

• The current utility of AI chatbots for drug information in pharmacovigilance requires further investigation.
• Additional studies across multiple pharmacovigilance centers are necessary to confirm or refute the value of this technology.
• Expert human oversight remains critical for ensuring accurate drug safety assessments.