Evaluation of ChatGPT-4 and Microsoft Copilot for Third-Molar Assessment on Panoramic Radiographs

To assess the performance of two AI chatbot assistants in identifying the presence and classifying the position of third molars on panoramic radiographs. A total of 114 third molars from 100 panoramic radiographs were evaluated consensually by three examiners and independently by two AI chatbot assistants (ChatGPT-4 and Microsoft Copilot). They were asked to provide descriptions regarding the orientation of the third molar's long axis, level of bone inclusion, space between the lower second molar and the mandibular ramus, and proximity of the third molar to the mandibular canal or maxillary sinus. Keywords generated by the AI chatbot assistants were compared to those used by the examiners and scored as 0 (incorrect), 0.5 (partially correct), or 1 (correct). Mean scores and standard deviations were calculated for each parameter and compared using the Wilcoxon test (α = 0.05). Mean scores across the four parameters ranged from 0.08 to 0.30 (SD = 0.42-0.44) for ChatGPT-4 and from 0.25 to 0.31 (SD = 0.42-0.47) for Microsoft Copilot. The only significant difference in performance between the AI chatbots was observed in the space between the lower second molar and ramus, in favor of Microsoft Copilot (p < 0.05). Overall performance scores were 0.22 (SD = 0.42) for ChatGPT-4 and 0.28 (SD = 0.46) for Microsoft Copilot. Furthermore, hallucinations such as classifying absent teeth were also observed. Both ChatGPT-4 and Microsoft Copilot demonstrate generally low performance in accurately identifying and classifying the position of third molars on panoramic radiographs.