Performance of large language models in urological decision support: a guideline-based comparative evaluation in...

Area of Science:

• Artificial Intelligence in Medicine
• Urology
• Clinical Decision Support

Background:

• Large language models (LLMs) show promise for clinical information support.
• Their consistency with subspecialty guidelines, especially in urolithiasis, needs evaluation.

Purpose of the Study:

• To assess the guideline concordance of four LLMs (GPT-4, GPT-4-turbo, Claude, Gemini) for urolithiasis clinical questions.
• To evaluate LLM performance in generating accurate, guideline-based responses.

Main Methods:

• 105 urolithiasis clinical questions were developed.
• Four LLMs generated responses in two sessions.
• Two urologists evaluated responses for accuracy and guideline adherence.

Main Results:

• All LLMs demonstrated high guideline adherence (mean scores 86.5-92.8%).
• Claude showed the highest correlation with expert ratings (r=0.94).
• No significant differences were found across models or clinical categories; high repeatability was observed.

Conclusions:

• LLMs, particularly Claude, provide reliable, guideline-consistent answers for urolithiasis.
• LLMs show potential as adjunctive tools for urological education and guideline support.
• Further real-world clinical validation is necessary.