Clinical safety, content coverage, and patient-centered language of AI responses to periodontal complaint-based...

This study aimed to evaluate the clinical safety, informational completeness, and patient-centered language of responses generated by four large language model (LLM)-based systems to standardized periodontal complaint-based queries. Seven standardized symptom-based periodontal queries were developed based on commonly reported patient complaints and submitted in Turkish to Copilot, Gemini, Claude, and ChatGPT. Responses were evaluated using a structured rule-based framework consisting of a Content Coverage Score (CCS), Risk of Harm Score (RHS), and Patient Language Score (PLS). Assessments were performed by two blinded human reviewers and one blinded AI-based evaluator. Human consensus, AI consensus, and combined evaluator scores were calculated. Agreement between evaluators was assessed using intraclass correlation coefficients (ICC), while human-AI differences and inter-model comparisons were analyzed using non-parametric statistical tests. Excellent agreement was observed between human reviewers, repeated AI evaluations, and human-AI consensus scores (ICC range: 0.911-0.925; p < 0.001). No significant difference was found between overall human and AI consensus scores (p = 0.985). Across the four AI systems, no statistically significant differences were observed in CCS or PLS scores in the human, AI, or combined evaluator analyses (all p > 0.05). PLS scores were generally high across models, indicating good linguistic accessibility for patients. No clearly harmful guidance was identified by the human reviewers. Overall, LLM-based systems generated clinically safe, reasonably comprehensive, and generally patient-accessible responses to common periodontal complaint-based queries. Although these systems may serve as supplementary sources of periodontal health information, they cannot replace individualized clinical evaluation and professional dental consultation.