Assessing ChatGPT Responses to Commonly Asked Questions About Hypoglossal Nerve Stimulation

This study aimed to explore the accuracy and comprehensiveness of 10 frequently asked questions posed to ChatGPT, an online chatbot, regarding obstructive sleep apnea (OSA) and hypoglossal nerve stimulation (HNS) therapy.

Ten questions were formulated after extensive literature review alongside the guidance of the senior author (R.N.), a board-certified otolaryngologist specializing in HNS therapy. We employed ChatGPT version 3.5 to answer each question, limiting responses to one paragraph. No follow-up queries were posed. The precision and thoroughness of responses were graded via the grading scale proposed by Mika et al. (within the context of hip arthroplasty) and the DISCERN criteria. Each response was graded by three independent, blinded reviewers (L.L., D.M., and R.N.).

On average, the responses produced by ChatGPT were satisfactory but typically required moderate clarification (inter-rater reliability [IRR] 0.533). Of the 10 responses, only one response was deemed excellent not requiring clarification. Two of the responses were deemed satisfactory requiring minimal clarification and the remaining seven responses were all deemed satisfactory requiring moderate clarification. No response was deemed unsatisfactory requiring substantial clarification. The average DISCERN score was 41.0 (IRR = 0.513), suggesting that the written health information was "average."

The responses produced by ChatGPT were fairly accurate, but often benefited from further clarification to ensure a comprehensive understanding. Ultimately, it seems that ChatGPT currently functions as a powerful resource to aid patients in further understanding topics related to OSA and HNS therapy, but should not replace direct consultation with knowledgeable healthcare providers.