Is ChatGPT-4 a Reliable Tool in Autoimmune Hepatitis?

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Gastroenterology

Background:

• Artificial intelligence (AI)-based chatbots present a novel approach for delivering personalized patient counseling in autoimmune hepatitis.
• Chat Generative Pretrained Transformer-4 (GPT-4) was evaluated for its accuracy, completeness, comprehensiveness, and safety in responding to patient inquiries.

Purpose of the Study:

• To assess the performance of GPT-4 in answering patient questions regarding autoimmune hepatitis.
• To evaluate the reliability and safety of AI chatbot responses in a clinical context.

Main Methods:

• 12 questions from 4 autoimmune hepatitis patients were categorized into diagnosis, quality of life, and medical treatment.
• 11 key opinion leaders rated GPT-4 responses using Likert scales for accuracy, safety, completeness, and comprehensiveness.

Main Results:

• Median scores indicated good comprehensibility but low reliability across evaluated domains.
• Accuracy and completeness were particularly low for post-diagnosis follow-up questions, though responses remained safe and comprehensive.
• Inter-rater agreement among experts was poor for most evaluated features, except for slight agreement on accuracy.

Conclusions:

• While AI chatbots demonstrate potential for patient engagement, their current lack of reliability hinders integration into clinical practice.
• Further research is essential to improve the dependability and safety of AI chatbots before widespread adoption in autoimmune hepatitis care.