Assessing the concordance between ChatGPT-4 and EAU guidelines indications in bladder cancer management: a real-world...

The integration of Artificial Intelligence (AI), particularly Large Language Models (LLMs) like ChatGPT, into healthcare is reshaping patient information-seeking behaviors. While prior studies have evaluated ChatGPT's performance across various urological domains, no research has yet addressed its ability to provide guideline-concordant recommendations for Bladder Cancer (BCa) management based on histopathological findings. This study aimed to evaluate the concordance between ChatGPT-4 and urologists' therapeutic recommendations for patients undergoing transurethral resection of bladder tumor (TURBT), using the European Association of Urology (EAU) guidelines as the reference.

We retrospectively analyzed 219 consecutive patients who underwent TURBT for BCa at a tertiary referral center between January and June 2024. Data collected included demographics, oncological history, and histopathology reviewed according to WHO 2022 standards. Patients were risk-stratified using the 2021 EAU Non Muscle-Invasive Bladder Cancer (NMIBC) risk model. Two urologists independently defined treatment plans per 2024 EAU guidelines. ChatGPT-4 (accessed April 2025) was queried using histopathological reports and previous BCa history. Recommendations from ChatGPT and urologists were compared for agreement, evaluated using Cohen's Kappa coefficient.

Among the 219 patients (mean age 72.36 years, 78.5% male), 42.9% had de novo BCa, 6.8% underwent II look TURBT, and 50.2% had recurrence. Histopathology revealed 68.5% pure urothelial carcinoma and 53.8% high-grade tumors. NMIBC risk classification identified 5.5% low, 19.6% intermediate, 32.9% high, and 9.1% very-high risk cases. Overall agreement between urologist and ChatGPT recommendations was poor (Cohen's K=-0.06). Stratified analyses showed moderate agreement for first-time BCa (K=0.23) and very high agreement in muscle-invasive BCa (MIBC) cases (K=0.88). In contrast, recurrent cases and II look TURBT showed low to negative concordance (K=-0.24 and K=-0.6, respectively). ChatGPT displayed a notably conservative attitude towards recommending early radical cystectomy, aligning with guidelines in only 3 of 21 cases.

Despite good readability and accessibility, ChatGPT-4's therapeutic recommendations based solely on histopathological reports lacked consistency with EAU guidelines, especially in complex scenarios such as recurrent disease or post-intravesical therapy management. Concordance was higher in straightforward cases like initial diagnosis or MIBC. These findings underscore the limitations of AI when clinical context is incomplete and highlight the irreplaceable role of clinician judgment. ChatGPT may serve as a supplementary tool for patient education but should not be used independently for clinical decision-making in BCa.