Comparative Evaluation of Popular Gen-AI Chatbots in Generating Patient Education Material on Pulmonary Artery...

Patient education significantly improves outcomes, especially in high-risk procedures. However, traditional educational resources often fail to address patient literacy and emotional needs adequately. Large language models like ChatGPT (OpenAI) and Gemini (Google) offer promising alternatives, potentially enhancing both accessibility and comprehensibility of procedural information. This study evaluates and compares the effectiveness of ChatGPT and Gemini in generating accurate, readable, and clinically relevant patient education materials (PEMs) for pulmonary artery catheter insertion.

A comparative, single-blinded study was conducted using structured validation methods using a common prompt for both gen artificial intelligence (AI) chatbots. AI-generated PEMs were assessed by board-certified anesthesiologists and intensivists. Face validity was determined using a 5-point Likert scale evaluating appropriateness, clarity, relevance, and trustworthiness. Content validity was measured by calculating content validity index. Accuracy and completeness were evaluated by a separate expert panel using a 10-point Likert scale. Readability and sentiment analysis were assessed via automated online tools.

Both chatbots achieved robust face and content validity (S-CVI = 0.91). ChatGPT scored significantly higher on accuracy [9.00 vs. 8.00; P = 0.021] and perceived trustworthiness, while Gemini outperformed in readability (Flesch Reading Ease score: 65 vs. 54; Flesch-Kincaid Grade Level: 7.58 vs. 8.64) and clarity. Both outputs maintained a neutral emotional tone.

AI chatbots show promise as innovative tools for patient education. By leveraging the strengths of both AI-driven technologies and human expertise, healthcare providers can enhance patient education and empower individuals to make informed decisions about their health and medical care involving complex clinical procedures.