Evaluation of the Reliability of AI-Based Large Language Models in Developing Orthodontic Treatment Plans

Background and aim Orthodontic treatment planning is a complex process requiring a detailed understanding of dental, skeletal, and soft tissue relationships. Traditionally, treatment decisions are made through clinical expertise and evidence-based guidelines. However, the recent evolution of AI, particularly large language models (LLMs), has warranted an evaluation of their capabilities in streamlining clinical workflows. The aim of this study was to evaluate the proficiency and effectiveness of AI-based LLMs, specifically OpenAI's ChatGPT-4o and Google's Gemini 2.0 Flash Experimental (free version), in generating orthodontic treatment plans based on real clinical cases. Materials and methods Ten published orthodontic case reports from reputed peer-reviewed journals were selected for the study and summarized into standardized clinical inputs, including patient age, occlusal relationships, skeletal and dental findings, and radiographic observations. These inputs were submitted to ChatGPT-4o and Gemini 2.0 Flash Experimental (free version) with prompts to generate extremely detailed, comprehensive treatment plans. The outputs were evaluated independently by two experienced orthodontists and one orthodontic resident using a four-point ordinal scale assessing clinical accuracy, completeness, and relevance of the treatment plan. Inter-rater reliability was assessed using Krippendorff's alpha. Results ChatGPT-4o produced treatment plans with higher clinical alignment and evaluator consensus, as indicated by Krippendorff's alpha (α = 0.935), while Gemini's plans showed greater variability and moderate agreement (α = 0.692). ChatGPT generated orthodontic treatment plans that incorporated more relevant clinical details and demonstrated stronger alignment with evidence-based standards, as assessed by the orthodontic reviewers. In contrast, Gemini generated treatment plans based on minimally accurate facts. Conclusion LLMs such as ChatGPT-4o and Gemini 2.0 Flash Experimental (free version) demonstrate potential as valuable complementary tools in orthodontic treatment planning, especially in routine cases, but do not appear to have the ability to replace clinical expertise.