A comparison of quality and readability of Artificial Intelligence chatbots in triage for head and neck cancer

Head and neck cancers (HNCs) are a significant global health concern, contributing to substantial morbidity and mortality. AI-powered chatbots such as ChatGPT, Google Gemini, Microsoft Copilot, and Open Evidence are increasingly used by patients seeking health information. While these tools provide immediate access to medical content, concerns remain regarding their reliability, readability, and potential impact on patient outcomes.

Responses to 25 patient-like HNC symptom queries were assessed using four leading AI platforms: ChatGPT, Google Gemini, Microsoft Copilot, and Open Evidence. Responses were evaluated using modified DISCERN criteria for quality and SMOG scoring for readability, with ANOVA and post hoc analysis conducted afterward.

Microsoft Copilot achieved the highest mean DISCERN score of 41.40 (95 % CI: 40.31 to 42.49) and the lowest mean SMOG reading levels of 12.56 (95 % CI: 11.82 to 13.31), outperforming ChatGPT, Google Gemini, and Open Evidence in overall quality and accessibility (p < .001). Open Evidence scored lowest in both quality averaging 30.52 (95 % CI: 27.52 to 33.52) and readability of 17.49 (95 % CI: 16.66 to 18.31), reflecting a graduate reading level.

Significant variability exists in the readability and quality of AI-generated responses to HNC-related queries, highlighting the need for platform-specific validation and oversight to ensure accurate, patient-centered communication.

Our study is a cross-sectional analysis that evaluates chatbot responses using established grading tools. This aligns best with level 4 evidence.