The understandability and actionability of AI-generated information for patients with chronic kidney disease

This study assessed the understandability, actionability, and overall quality of Artificial Intelligence (AI)-generated responses to frequently posed questions related to chronic kidney disease (CKD), compared to expert information.

A quantitative analysis to compare the quality of AI-generated responses and webpages containing information for patients with CKD was performed. Frequently searched keywords for CKD were entered into the ChatGPT 3.5, Copilot, and Gemini databases. Online patient education materials on CKD created by experts were also collected via Google searches. The Japanese version of the Patient Education Materials Assessment Tool was used to evaluate the understandability and actionability of the information.

A total of 180 AI responses and 88 items of expert information were incorporated. There was no significant differences between AI responses and expert information in terms of understandability (mean±SD: AI vs expert, 67.2%±14.0% vs 63.4%±15.9 %, P = 0.06). For actionability, expert information scored higher than AI responses (mean±SD: AI vs expert, 27.9%±16.2% vs 37.1%±27.6%, P < 0.01). The AI responses excelled in simplicity of writing, clarity of purpose, and use of numbers. However, AI responses lacked logic in information and specificity in instructions for action. Gemini outperformed ChatGPT and Copilot in both understandability and actionability (71.6​​​​​​% vs 63.7% and 66.2%, P = 0.01; 38.3% vs 20.7% and 24.7%, P < 0.01).

The AI responses showed comparable understandability to expert information, but outperformed expert information in terms of text readability. However, challenges have arisen regarding the actionability, flow, and comprehensiveness of AI responses.

Although AI tools can simplify medical terminology, healthcare professionals should enhance AI-generated CKD information to ensure actionability when disseminating it to patients.