Evaluating ChatGPT Responses on Atrial Fibrillation for Patient Education

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Education Technology
• Digital Health

Background:

• ChatGPT, a popular AI chatbot, is known for conversational fluency but can generate inaccurate information.
• Evaluating the accuracy and educational suitability of AI-generated content is crucial for patient safety and understanding.
• Atrial fibrillation (AF) patient education requires clear, accurate, and accessible information.

Purpose of the Study:

• To assess the accuracy and readability of ChatGPT responses to atrial fibrillation (AF) patient education questions.
• To determine if ChatGPT can tailor response complexity to different user prompts (patient vs. physician).
• To evaluate the rate of reference provision for AI-generated medical information.

Main Methods:

• ChatGPT was prompted with 16 frequently asked questions about AF from the American Heart Association.
• Four prompt forms were used: no prompt, patient-friendly, physician-level, and statistics/references.
• Responses were scored for accuracy (incorrect, partially correct, correct, perfect) and Flesch-Kincaid grade level.

Main Results:

• Over 85% of ChatGPT responses were rated as correct, with only 1.6% incorrect.
• Perfect responses (correct with references) were rare (4.7%), even when specifically requested.
• Patient-friendly prompts resulted in significantly lower mean grade levels compared to other forms.
• Response accuracy did not significantly differ by prompt form, but the rate of perfect responses did.

Conclusions:

• ChatGPT demonstrates significant potential as an accurate informational resource for patient education in healthcare.
• The AI's ability to adapt response complexity to user input is a key advantage for personalized education.
• Continuous monitoring and advancement of AI are necessary to maximize benefits and mitigate risks in AI-driven healthcare education.