[Evaluation of ChatGPT's responses to symptom-oriented questions in otolaryngology]

Introduction: Chat Generative Pre-Trained Transformer (ChatGPT) is a recently developed artificial intelligence (AI)-based language model that has become an increasingly common source of health-related information due to its accessibility. However, there is limited evidence regarding the accuracy and reliability of its responses. Objective: This study aimed to assess ChatGPT’s usability in otolaryngology by analyzing its answers to common patient questions. Method: 24 patient-oriented questions were created across 10 otolaryngological disease categories and submitted sequentially to ChatGPT version 4.0. The generated responses were evaluated from a linguistic perspective and 6 board-certified otolaryngologists using a three-point scale: incorrect (1), correct but incomplete (2), and correct (3). Results: Language evaluations indicated that the responses were generally clear, well-structured, and of good quality for further medical evaluations. ChatGPT’s medical accuracy scores ranged from 2.00 to 2.83. Three specific questions (Q4, Q9, Q24) received significantly lower ratings. Four raters (R1, R2, R4, R6) showed strong agreement in their evaluations, while significant differences emerged between the scores of R3 and R5 (p<0.001). Conclusion: ChatGPT’s responses in otolaryngology were coherent and well-structured, but the accuracy of medical content varied by topic. While the tool may be beneficial for basic patient education, it is not currently reliable enough to support clinical decision-making. Future development of validated, specialty-specific artificial intelligence systems based on trustworthy medical sources will be crucial for safe implementation in healthcare. Orv Hetil. 2025; 166(42): 1666–1674.