Evaluating AI Chatbot Responses to Postkidney Transplant Inquiries

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Transplantation Medicine

Background:

• Post-kidney transplantation care involves complex patient inquiries.
• Patients increasingly turn to online resources, including AI chatbots, for health information.
• Evaluating the accuracy and usability of AI-generated health information is crucial.

Purpose of the Study:

• To assess the performance of AI chatbots (ChatGPT 4.0, Claude 3.0, Gemini Pro) and Google in answering common post-kidney transplant questions.
• To evaluate response quality, understandability, actionability, and readability of AI-generated content.
• To compare the capabilities of different AI models and traditional search engines for patient-specific medical queries.

Main Methods:

• A comprehensive list of 127 post-kidney transplant questions was compiled using Google and Bing.
• AI chatbots and Google provided responses to these questions.
• Response quality was rated using a 5-point Likert scale.
• Understandability and actionability were assessed using the Patient Education Materials Assessment Tool (PEMAT).
• Readability was measured using Flesch Reading Ease and Flesch-Kincaid Grade Level metrics.
• Statistical analysis was performed using the Kruskal-Wallis test in SPSS.

Main Results:

• AI chatbots and Google provided high-quality responses (median Likert score: 4/5) and good understandability (median PEMAT score: 72.7%).
• However, responses demonstrated poor actionability (median PEMAT score: 20%) and challenging readability (median Flesch Reading Ease: 22.1; median Flesch-Kincaid Grade Level: 14.7).
• Claude 3.0 offered the most reliable answers, while ChatGPT 4.0 provided the most comprehensible responses.
• Google's performance was not superior to any of the AI chatbots across all evaluated metrics.

Conclusions:

• Current AI chatbots and search engines provide generally accurate but not easily actionable information for post-kidney transplant patients.
• The readability level of AI-generated responses is often too high for the general patient population.
• Further development is needed to enhance the actionability and improve the readability of AI-generated health information for transplant recipients.