Orthodontic knowledge assessment: A comparison of five AI Chatbots.

Area of Science:

• Artificial Intelligence
• Orthodontics
• Medical Informatics

Background:

• The increasing use of AI in healthcare necessitates evaluating its accuracy in specialized fields.
• Orthodontics, a complex dental specialty, requires precise information for effective patient care and education.

Purpose of the Study:

• To assess the performance and accuracy of five distinct AI chatbots in delivering orthodontic information.
• To compare AI chatbot responses against established orthodontic knowledge and dental student performance.

Main Methods:

• Eighty multiple-choice questions (MCQs) from orthodontic exam materials were used to test five AI chatbots: ChatGPT, DeepSeek, Gemini, MedgebraGPT, and Meta AI.
• Chatbot accuracy was evaluated by comparing their answers to correct answers and dental student responses.
• Item-difficulty index scores were calculated for AI responses, with separate analyses for dependent and independent MCQs.

Main Results:

• DeepSeek provided the highest proportion of correct answers, whereas MedgebraGPT provided the lowest.
• Gemini and DeepSeek showed performance improvement in a second round, with moderate and substantial agreement, respectively.
• ChatGPT and Meta AI performance did not improve in the second round; inter-rater reliability between AI and student performance was low and not statistically significant.

Conclusions:

• DeepSeek demonstrated the highest accuracy in providing orthodontic information among the tested AI chatbots.
• MedgebraGPT exhibited the lowest accuracy, highlighting significant variability in AI performance.
• While AI chatbots possess some understanding of orthodontics, their accuracy and consistency require further improvement for reliable use.