Let's Have a Chat: How Well Does an Artificial Intelligence Chatbot Answer Clinical Infectious Diseases...

Area of Science:

• Artificial Intelligence in Healthcare
• Pharmacotherapy
• Infectious Diseases

Background:

• The quality of ChatGPT (version 3.5) responses for infectious diseases (ID) pharmacotherapy is not well-established.
• Subject matter experts (SMEs) in ID pharmacotherapy were surveyed to evaluate ChatGPT's response quality.

Purpose of the Study:

• To assess the usefulness, correctness, completeness, and safety of ChatGPT (version 3.5) responses to ID pharmacotherapy questions.
• To determine if ChatGPT can serve as a reliable tool for ID pharmacotherapy information.

Main Methods:

• One hundred ID pharmacotherapy questions were input into ChatGPT (version 3.5).
• ID pharmacist SMEs rated the responses on a 0-10 scale for correctness, completeness, and safety.
• Interrater reliability was established using Fleiss multirater kappa and intraclass correlation coefficients.

Main Results:

• Only 41.8% of ChatGPT responses were deemed useful by SMEs.
• Median ratings for correctness, completeness, and safety were 7, 5, and 8, respectively.
• Responses were generally rated as correct and safe but frequently lacked completeness.

Conclusions:

• ChatGPT (version 3.5) responses were considered useful in less than 50% of cases for ID pharmacotherapy.
• The incompleteness of responses suggests that ChatGPT cannot currently replace the expertise of an ID pharmacist.