Accuracy and Readability of Artificial Intelligence Chatbot Responses to Vasectomy-Related Questions: Public Beware

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Urology

Background:

• Large-language model chatbots like ChatGPT are increasingly used by the public for medical information.
• There is limited research on the reliability of these AI tools for patient-specific medical queries.
• Assessing AI accuracy and readability is crucial for safe public health information dissemination.

Purpose of the Study:

• To evaluate the accuracy and readability of AI chatbots in responding to patient questions about vasectomy.
• To compare the performance of five popular AI platforms (ChatGPT, Bard, Bing, Perplexity, Claude) on urological queries.
• To determine the suitability of AI chatbots as reliable sources for medical information.

Main Methods:

• Fifteen vasectomy-related questions were posed to five AI chatbots (ChatGPT, Bard, Bing, Perplexity, Claude).
• Responses were assessed for accuracy by urology experts using a 6-point Likert scale against American Urological Association guidelines.
• Readability was measured using Flesch Reading Ease Scores (FRES) and Flesch-Kincaid Grade Levels (FKGL).

Main Results:

• ChatGPT demonstrated the highest average accuracy (5.04/6), followed by Bing (4.91), Claude (4.65), Bard (4.43), and Perplexity (4.41).
• All chatbots scored at least "somewhat accurate" on average.
• Google Bard had the best readability (FRES 49.67, FKGL 10.1), while ChatGPT had the lowest FRES (30.4) and highest FKGL (14.2).

Conclusions:

• Major AI chatbots exhibit comparable accuracy in providing medical information on vasectomy.
• Significant differences exist in the readability of AI-generated responses, impacting patient comprehension.
• AI chatbots can serve as a "somewhat accurate" source of medical information, but varying readability necessitates careful consideration for public use.