Thyro-GenAI: A Chatbot Using Retrieval-Augmented Generative Models for Personalized Thyroid Disease Management.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large language models (LLMs) show promise for healthcare but suffer from inaccuracies and hallucinations.
• Retrieval-augmented generation (RAG) can mitigate these issues by incorporating external knowledge.
• Thyroid disease management requires accurate and reliable clinical reasoning.

Purpose of the Study:

• To develop and evaluate a RAG-based chatbot, Thyro-GenAI, for clinical decision support in thyroid disease.
• To compare the performance of Thyro-GenAI against leading service LLMs in generating responses to clinical questions.
• To assess the quality of responses and references provided by different AI models.

Main Methods:

• Developed Thyro-GenAI by integrating a LLM with a database of thyroid disease textbooks and guidelines.
• Presented personalized clinical questions about thyroid disease to Thyro-GenAI, ChatGPT-4o, Perplexity AI, and Claude 3.5 Sonnet.
• Evaluated response and reference quality by three blinded thyroid specialists using inverse-weighted mean ranks.

Main Results:

• Thyro-GenAI achieved the highest inverse-weighted mean rank for overall response quality (3.0).
• Thyro-GenAI secured the second-highest rank for overall reference quality (3.1).
• Service LLMs (ChatGPT-4o, Perplexity AI, Claude 3.5 Sonnet) received lower quality rankings.

Conclusions:

• Thyro-GenAI provides more reliable, patient-specific clinical reasoning with fewer hallucinations than general service LLMs.
• The RAG approach enhances the safety and validity of AI-generated medical information.
• Thyro-GenAI has the potential to significantly support frontline physicians in managing thyroid disease.