Assessing ChatGPT's Responses to Otolaryngology Patient Questions.

Area of Science:

• Artificial Intelligence in Medicine
• Otolaryngology
• Digital Health

Background:

• Large language models like ChatGPT are increasingly used for health information.
• Assessing AI's accuracy and safety in specialized medical fields is crucial.
• Otolaryngology patient queries present unique challenges for AI interpretation.

Purpose of the Study:

• To evaluate ChatGPT's performance in answering otolaryngology patient questions.
• To assess the accuracy, comprehensiveness, and patient safety of AI-generated medical advice.
• To determine the suitability of ChatGPT for integration into otolaryngology healthcare.

Main Methods:

• Cross-sectional study using patient questions from Reddit's r/AskDocs.
• ChatGPT (GPT-3.5) responses were reviewed by 5 otolaryngologists.
• Evaluation criteria included accuracy, comprehensiveness, and bedside manner; dangerous responses were identified.

Main Results:

• ChatGPT responses averaged 414.93 words, with accuracy 3.76/5 and comprehensiveness 3.59/5.
• Bedside manner/empathy scored high (4.28/5), but longer responses did not improve ratings.
• Five responses were flagged as potentially dangerous; higher difficulty correlated with lower comprehensiveness.

Conclusions:

• ChatGPT demonstrates promise but has limitations in accuracy and comprehensiveness for otolaryngology queries.
• Potentially dangerous responses highlight the need for caution when using AI for medical advice.
• Responsible AI integration requires rigorous performance and safety assessments.