Urological Cancers and ChatGPT: Assessing the Quality of Information and Possible Risks for Patients

OpenAI has created ChatGPT, an artificial intelligence language model that has gained considerable recognition for its capacity to produce text responses resembling human language. Consequently, this study seeks to evaluate the effectiveness of ChatGPT's responses in addressing publicly accessible queries related to prostate, kidney, bladder, and testicular cancers.

A comprehensive compilation of frequently asked questions (FAQs) pertaining to prostate, bladder, kidney, and testicular cancers was gathered from diverse sources. Additionally, the recommendations outlined in the European Association of Urology (EAU) 2023 Guideline Oncology were consulted. The chosen questions for evaluation were presented to the ChatGPT 4.0 premium version. The quality of ChatGPT responses was appraised using the global quality score (GQS). Each ChatGPT response was independently reviewed by a panel of physicians, who assigned a GQS score to assess its overall quality.

For prostate cancer, 64.6% of the questions had a GQS score of 5, compared to 62.9 % for bladder, 68.1% for kidney, and 63.9% for testicular cancers, whereas none of the responses had a GQS score of 1. Meanwhile, the category with the lowest proportion of responses, with a GQS score of 5 for each disease, was prognosis and follow-up. The mean GQS score of the answers given to EAU guideline questions was statistically significantly lower than the average score of the answers given to FAQs.

ChatGPT is a valuable tool for addressing general inquiries regarding urological cancers, boasting commendable accuracy rates. Nonetheless, its performance in responding to questions aligned with the EAU guideline was deemed unsatisfactory.