Enhancing clinicians' trust in large language models via transparent source attribution: A randomized controlled...

Area of Science:

• Artificial Intelligence in Medicine
• Uro-oncology Clinical Decision Support

Background:

• Large language models (LLMs) in urology and oncology face limitations due to outdated data and lack of source transparency.
• These limitations hinder clinical reliability and adoption of AI tools in healthcare.

Purpose of the Study:

• To develop and evaluate UroBot, a urology-specific chatbot enhanced with retrieval-augmented generation (RAG).
• To compare UroBot's performance against ChatGPT in providing accurate and verifiable clinical recommendations.

Main Methods:

• UroBot was developed integrating RAG for in-line references and source text previews.
• A randomized controlled reader study compared UroBot and ChatGPT across ten uro-oncological cases.
• Thirty urologists assessed recommendation correctness, source verifiability, and trust.

Main Results:

• UroBot demonstrated superior recommendation correctness (73% vs. 50%), source attribution (74% vs. 30%), and verifiability (84% vs. 35%) compared to ChatGPT.
• Clinicians consistently preferred UroBot for accuracy, verifiability, and trust.
• ChatGPT provided vague or incorrect citations, with 28% non-existent/outdated and 83% lacking specific sections, while UroBot achieved precise citation alignment.

Conclusions:

• Combining LLMs with RAG significantly enhances source attribution and verifiability over conventional LLMs.
• This RAG-enhanced approach provides reliable and up-to-date clinical decision support.
• The methodology is transferable across medical subspecialties.