Accuracy, readability, and understandability of European Association of Urology guidelines bot for Sexual and...

Recently, the European Association of Urology (EAU) Guidelines presented an official Bot to assist urologists during Guidelines navigation. However, up to date no external validation is available. Aim: To assess accuracy, completeness, and clarity of the Guidelines Bot for Sexual and Reproductive Health.

A total of 228 questions based on the EAU Sexual and Reproductive Health Guidelines recommendations were developed. Each question was inputted to the EAU Guidelines Bot and the response was reviewed by two expert uro-andrologists. Discrepancies were resolved by discussion with a third expert. Results were further stratified per grade of recommendation. Outcomes: Evaluate the rate of accurate, complete, and clear answers to guidelines-related questions using a 5-point Likert scale and the impact of the grade of recommendation on the quality of the answer.

Overall, 228 questions were developed. In terms of accuracy 224/228 (98.3%) were defined as accurate (score-4-5), 2/228 (0.9%) presented a fair accuracy (score = 3) while 2/228 (0.9%) were deemed not accurate (score 1-2). In terms of completeness, 223/228 (97.8%) were defined as complete (score-4-5), 2/228 (0.9%) presented a fair completeness (score 3), while 3/228 (1.3%) were deemed not complete. Finally in terms of clarity, 225/228 (98.7%) were defined as clear (score-4-5), 2/228 (0.9%) presented a fair clarity (score 3) and 0/228 were not clear. When comparing strong and weak recommendations, no differences were recorded.

The EAU Guidelines Bot may serve as a reliable clinical decision support tool for urologists seeking rapid, evidence-based guidance on sexual and reproductive health management.

This is the first external evaluation of the EAU Guidelines Bot. Our results suggest a significant improvement in terms of reliability when compared to general AI tools. However, our queries were straightforward and developed directly from guideline recommendations and results might not apply to complex real-world clinical scenarios.

EAU Guidelines Bot represents an accurate and reliable tool for Sexual and Reproductive Health Guidelines navigation, but further validation is required to evaluate its applicability in clinical practice.