Quality assessment of patient-facing urologic telesurgery content using validated tools

Area of Science:

• Artificial Intelligence in Medicine
• Digital Health
• Medical Informatics

Background:

• Increasing accessibility of AI chatbots necessitates rigorous assessment of medical information accuracy.
• Urologic telesurgery is a complex topic patients may explore using AI tools.
• This study evaluates AI chatbots for patient-facing urologic telesurgery information.

Purpose of the Study:

• To compare the quality and understandability of information provided by ChatGPT and Google Gemini on urologic telesurgery.
• To assess the actionability of AI-generated responses for patient decision-making.
• To identify the strengths and limitations of current AI chatbots in delivering medical information.

Main Methods:

• Generated 19 questions on urologic telesurgery based on AUA and ERUS guidelines.
• Administered questions to non-paid versions of ChatGPT 4o and Google Gemini 2.5.
• Evaluated responses using DISCERN (quality) and PEMAT-P (understandability, actionability) tools by three independent reviewers.

Main Results:

• Google Gemini achieved higher mean DISCERN scores for quality across most domains compared to ChatGPT.
• Both chatbots provided understandable information, with PEMAT-P scores consistently above 70%.
• Actionability scores were uniformly low for both platforms, with only Gemini meeting the threshold in the prospective domain.

Conclusions:

• ChatGPT and Google Gemini offer relevant and understandable patient-facing information on urologic telesurgery.
• Gemini demonstrated superior quality and source provision.
• Neither AI chatbot currently provides sufficiently actionable responses for independent patient decision-making.