Performance of artificial intelligence chatbots compared with young academic urologists in reconstructive urology

Artificial intelligence (AI) is increasingly used in surgery, but its role in reconstructive urology remains insufficiently studied. The aim of this work was to evaluate the theoretical knowledge of several AI platforms and compare their performance with that of experts from the Young Academic Urologists (YAU) group of the European Association of Urology.

This cross-sectional comparative study included 11 YAU experts and 4 AI platforms. Thirty-one multiple-choice questions from Chapter 82 of the Campbell-Walsh-Wein Urology Review, 3rd edition (2020), were used. Total scores and thematic scores were compared.

The mean score of YAU experts was 14 of 31 (SD 5.28). AI platforms achieved a mean score of 15 of 31 (SD 5.53), with no significant difference between groups (P=0.802). ChatGPT-4 achieved 24 correct answers and was surpassed by only one human expert. Subgroup analyses showed no significant differences between YAU experts and AI platforms for anatomy (P=0.272), physiology (P=0.923), therapeutic management (P=0.51) and diagnosis (P=0.611).

AI platforms, particularly ChatGPT-4, achieved high scores on MCQs in reconstructive urology, and no statistically significant difference was observed compared with YAU experts. These findings support a potential role for AI in educational settings and for answering well-defined theoretical questions, but their use alone is not suitable for clinical reasoning. Further validation on real clinical cases is required before considering clinical implementation.