Performance of ChatGPT in providing patient information about upper tract urothelial carcinoma

The aim was to evaluate ChatGPT generated responses to patient-important questions regarding upper tract urothelial carcinoma (UTUC).

Fifteen common inquiries asked by patients regarding UTUC were assigned to 4 categories: general information; symptoms and diagnosis; treatment; and prognosis. These questions were entered into ChatGPT and its responses were recorded. In every answer 5 criteria (adequate length, comprehensible language, precision in addressing the question, compliance with European Association of Urology guidelines and safety of the response for the patient) were assessed by the urologists using a numerical scale of 1-5 (a score of 5 being the best).

Sixteen questionnaires were included. A score of five was assigned 336 times (28.0%); 4 - 527 times, (43.9%); 3 - 268 times (22.3%); 2 - 53 ti- mes (4.4%); and 1 - 16 times (1.3%). The average overall score was 3.93. Responses to each question received average scores within the range 3.34-4.18. Answers regarding "general information" were graded the highest - mean score 4.14. Artificial intelligence scored the lowest in the "treatment" category - mean score 3.68. A mean score of 4.02 was given for the safety of the response. However, a few urologists considered several answers as unsafe for the patient, by grading them 1 or 2 in this criterion.

ChatGPT does not provide fully adequate information on UTUC, and inquiries regarding treatment can be misleading for the patients. In particular cases, patients might receive potentially unsafe answers. However, ChatGPT can be used with caution to provide basic information regarding epidemiology and risk factors of UTUC.