Can Patients With Urogenital Cancer Rely on Artificial Intelligence Chatbots for Treatment Decisions?

Area of Science:

• Urology
• Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Patients increasingly use online resources, including AI, for health information.
• The reliability of AI-generated medical information, particularly for complex conditions like cancer, requires thorough evaluation.

Purpose of the Study:

• To assess the reliability and quality of information provided by artificial intelligence chatbots (AICs) regarding urogenital cancer treatments.
• To compare the performance of different AICs in delivering accurate and understandable cancer treatment information.

Main Methods:

• Investigated frequently searched urogenital cancer treatment terms using Google Trends.
• Queried three leading AICs (ChatGPT, Gemini, Copilot) with these terms.
• Evaluated AIC responses using DISCERN scores, PEMAT-P for understandability and actionability, and Coleman-Liau index for readability.

Main Results:

• ChatGPT and Gemini provided moderate quality information (DISCERN scores 41, 42), while Copilot's quality was low (35).
• Understandability scores (PEMAT-P) were low across all AICs (40%).
• Actionability scores were moderate for Gemini (60%) but low for others (40%). Readability was consistently above college level.

Conclusions:

• Patients will likely continue using AICs for cancer treatment information due to accessibility.
• AICs currently provide moderate-quality, low-reliability information that is difficult to read and lacks stage-specific treatment options.
• Critical evaluation of AIC-generated health information is essential for patient safety and informed decision-making.