Incorporating information retrieval into AI chatbots for patient education on thyroid eye disease

Area of Science:

• Ophthalmology and Artificial Intelligence
• Medical Informatics
• Patient Education Technologies

Background:

• Thyroid eye disease (TED) patient questions cover diverse clinical and non-clinical topics.
• Evaluating AI chatbot performance is crucial for reliable patient information dissemination.
• Existing AI models vary in their suitability for medical inquiries.

Purpose of the Study:

• To compare the performance of general-purpose, retrieval-augmented, and medicine-specific AI chatbots in answering common TED patient questions.
• To assess AI responses based on clinical accuracy and patient-centered communication metrics.

Main Methods:

• A cross-sectional study synthesized 15 representative TED patient questions from online forums.
• Three AI models (ChatGPT, ChatGPT-RAG, OpenEvidence) generated responses.
• Oculoplastic surgeons and medical students evaluated responses for clinical and non-clinical metrics, respectively.

Main Results:

• The specialized medical AI (OpenEvidence) significantly outperformed general AI (ChatGPT) in clinical accuracy and relevance.
• General AI (ChatGPT) demonstrated superior performance in empathy, understandability, and actionability for non-clinical aspects.
• Retrieval-augmented generation (ChatGPT-RAG) showed intermediate performance across both clinical and non-clinical evaluations.

Conclusions:

• Specialized medical AI models offer greater clinical precision for TED inquiries.
• General-purpose AI models provide better patient communication and accessibility.
• Future AI development should integrate clinical accuracy with effective patient communication for optimal educational tools.