Related Experiment Video
Updated: Jan 14, 2026

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling
Benjamin Krichevsky1,2, Stefan Engeli3, Stefanie M Bode-Böger4
1Hannover Medical School, Institute for General Practice and Palliative Care, Hannover, Germany.
Aims:
To assess the utility of the artificial intelligence (AI) chatbot ChatGPT (openly available version 3.5) in responding to real-world pharmacotherapeutic queries from healthcare professionals.
Methods:
Three independent and blinded evaluators with different levels of medical expertise and professional experience (beginner, advanced, and expert) compared AI chatbot- and physician-generated responses to 70 real-world pharmacotherapeutic queries submitted to the clinical-pharmacological drug information centre of Hannover Medical School between June and October 2023 with regard to quality of information, answer preference, answer correctness and quality of language. Inter-rater reliability was assessed with Krippendorff's alpha. Two separate investigators not otherwise involved in the conduct or analysis of the study selected the top three clinically relevant errors in chatbot- and physician-generated responses.
Results:
All three evaluators rated the quality of information of physician-generated responses higher than the quality of information of AI chatbot-generated responses and, accordingly, thought that the physician-generated responses were better than the chatbot-generated responses (answer preference). All evaluators detected factually wrong information more frequently in chatbot-generated responses than in physician-generated responses. Although the beginner and expert evaluators rated the quality of language of physician-generated responses higher than the quality of language of chatbot-generated responses, there was no significant difference according to the advanced evaluator.
Conclusions:
ChatGPT's responses to real-world pharmacotherapeutic queries were substantially inferior compared to conventional physician-generated responses with regard to quality of information and factual correctness. Our study suggests that to date it must be strongly cautioned against the use of ChatGPT in pharmacotherapy counselling.
Related Concept Videos
Effect of Hepatic Disease on Pharmacokinetics: Dose Adjustments Due to Hepatic Impairment
Effect of Hepatic Disease on Pharmacokinetics: Pathophysiologic Assessment and Liver Function Test
Drug Administration and Therapy Phases: Overview
The pharmaceutical phase focuses on leveraging the physicochemical properties of the drug to design and manufacture an effective product. Variants include orally administered tablets or capsules, topical creams or ointments, and parenteral-delivery solutions or emulsions.
The pharmacokinetic phase...
Issues And Trends In Healthcare Delivery System
Cost Containment
Payment for healthcare services has historically promoted adoption of costly and often unnecessary or inefficient...
Bioequivalence of Drugs: Drugs with Multiple Indications
Effect of Hepatic Disease on Pharmacokinetics: Active Drug, Metabolite and Fraction of Metabolized Drug
