Accuracy of large language models in answering urological questions on lower urinary tract symptoms: comparison with...

The aim of the study was to evaluate the accuracy of responses from three common artificial intelligence (AI) tools - ChatGPT, Claude, and DeepSeek - to patient enquiries regarding lower urinary tract symptoms (LUTS), comparing these responses to European Association of Urology (EAU) 2025 guidelines. As patients increasingly turn to large language models (LLMs), it is crucial to assess their reliability.

Prospective face-to-face and telephone general urology clinics were conducted between April and May 2025, during which consented patients were asked to state the questions they would submit to an AI tool if they were to enquire about their LUTS. These questions were submitted to ChatGPT (GPT-4), Claude (Sonnet 4.0), and DeepSeek (V3), and the responses were summarised and compared against the EAU guidelines. Each response was categorised as correct, missing key elements from the guidelines, or incorrect/misleading.

Sixteen patients participated in the study, and following removal of duplicate questions, a total of 13 were included for analysis. These questions covered symptom causation, diagnostic workup and management of LUTS. All models provided correct information in 92.3% of their responses when compared to the EAU guidelines. However, 92.3-100% of answers omitted key elements from the guidelines, and 30.8-92.3% contained incorrect or misleading content.

Current LLMs provide readily accessible guidance on LUTS; however, their unsupervised use in clinical decision-making remains a concern and may be considered premature.