Evaluating ChatGPT as a Patient Education Tool for COVID-19-Induced Olfactory Dysfunction

Area of Science:

• Otolaryngology
• Artificial Intelligence in Medicine
• Digital Health

Background:

• Persistent olfactory dysfunction (OD) following COVID-19 can lead to significant physical and psychological distress.
• Artificial intelligence (AI) chatbots like ChatGPT are emerging as tools for patient education.
• The quality of AI-generated medical information requires careful evaluation, especially for conditions like COVID-19-induced OD.

Purpose of the Study:

• To assess the quality of responses generated by ChatGPT (GPT-4) regarding COVID-19-induced olfactory dysfunction (OD).
• To evaluate how different prompting strategies influence the accuracy and readability of ChatGPT's answers.
• To determine the suitability of ChatGPT as a patient education resource for COVID-19 OD.

Main Methods:

• A quantitative observational study querying ChatGPT (GPT-4) with 30 identical questions about COVID-19 OD.
• Responses were generated using four prompt types: patient-focused, eighth-grade level, with references, and no prompt.
• Four rhinologists scored answer accuracy using the Global Quality Score (GCS), and Flesch-Kincaid grade level was calculated.

Main Results:

• 75.8% of all ChatGPT responses (n=480) achieved a 'good' or 'excellent' quality score (GCS ≥ 4).
• Prompting significantly influenced response quality (P < .0001) and grade level, with patient-focused and eighth-grade prompts yielding lower levels.
• Average response grade levels (14.06-14.33) exceeded recommendations for patient information, even with specific prompts.

Conclusions:

• ChatGPT generally provides appropriate answers regarding COVID-19 OD, with quality influenced by prompting.
• The AI's response grade level often surpasses recommended levels for patient comprehension.
• ChatGPT holds potential as a supplementary patient education tool for COVID-19 OD, complementing traditional physician-patient interactions.