Evaluating large language models as patient education tools for inflammatory bowel disease: A comparative study

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Gastroenterology

Background:

• Inflammatory Bowel Disease (IBD) presents a significant global health challenge, increasing the need for effective patient education.
• Large Language Models (LLMs) offer a promising avenue for disseminating health information to patients.
• The efficacy of LLMs in providing accurate and understandable IBD information requires thorough investigation.

Purpose of the Study:

• To evaluate the utility of three leading LLMs (ChatGPT-4.0, Claude-3-Opus, Gemini-1.5-Pro) as a reference for patients with IBD.
• To compare the performance of these LLMs in answering patient-specific IBD queries.

Main Methods:

• Gastroenterology experts formulated 15 common patient questions concerning IBD.
• LLM responses were assessed by medical experts for accuracy, comprehensibility, and relevance using a Likert scale.
• Patients evaluated the comprehensibility of LLM-generated answers.
• Readability assessments were conducted on the provided information.

Main Results:

• All evaluated LLMs demonstrated satisfactory accuracy, comprehensibility, and completeness for basic IBD information (definitions, symptoms, diagnostics).
• Performance varied significantly for complex medical advice, including medication side effects, diet, and complication risks.
• Claude-3-Opus exhibited superior readability compared to ChatGPT-4.0 and Gemini-1.5-Pro.

Conclusions:

• LLMs hold potential as valuable educational tools for IBD patients.
• Discrepancies in LLM performance highlight the need for model optimization and specialized development.
• Ensuring the accuracy and safety of LLM-generated medical information is paramount for patient care.