Can AI chatbots accurately answer patient questions regarding vasectomies?

Area of Science:

• Urology
• Artificial Intelligence in Healthcare
• Medical Information Dissemination

Background:

• Artificial intelligence (AI) is transforming healthcare, yet its efficacy in urology remains understudied.
• Limited research exists on AI's accuracy for common urological procedures like vasectomy.
• Patient access to reliable information on vasectomies via AI tools is a growing concern.

Purpose of the Study:

• To assess the accuracy and clarity of AI chatbot responses to frequently asked questions about vasectomy.
• To compare the performance of three leading AI chatbots (ChatGPT, Bard, Bing) in providing vasectomy information.
• To evaluate AI chatbots as a potential resource for patients seeking information on vasectomy procedures.

Main Methods:

• Ten common vasectomy-related questions were posed to ChatGPT, Bard, and Bing.
• Responses were critically analyzed by three urologists using a 1-4 grading scale for clarity, accuracy, and evidence-based content.
• Statistical analysis was performed to compare the performance of the AI models.

Main Results:

• ChatGPT achieved the highest average rating (1.367), significantly outperforming Bard (2.167) and Bing (1.800).
• ChatGPT provided significantly more satisfactory answers than both Bard (p=0.00005) and Bing (p=0.03988).
• The performance difference between Bard and Bing was not statistically significant (p=0.09651).

Conclusions:

• AI chatbots, particularly ChatGPT, can provide largely accurate information regarding vasectomies.
• These AI tools represent a reasonable resource for patients seeking preliminary information about the vasectomy procedure.
• Further research is warranted to explore AI's role in patient education for urological procedures.