How Accurate Are the Responses of 4 AI Chatbots to Orthognathic Surgery Questions?

This study aimed to evaluate and compare the accuracy, reliability, and comprehensibility of information provided by 4 artificial intelligence (AI)-based language models (ChatGPT-4, Google Gemini, Microsoft Copilot, and DeepSeek-v3) for orthognathic surgery.

A cross-sectional content analysis was carried out to evaluate the responses generated by ChatGPT-4, Gemini, Copilot, and DeepSeek-v3. A total of 118 questions covering 12 domains related to orthognathic surgery were formulated, and the AI-generated answers were systematically assessed. A 5-point Likert scale was used to independently score the responses. Descriptive statistics were used. The Fisher exact test was applied to examine relationships between categorical variables when the expected value was <5. All analyses were performed by the IBM SPSS 27 program.

Significant differences were observed among the AI models ( P =0.022). DeepSeek-v3 demonstrated the highest proportion of objectively true responses (87.3%), outperforming Gemini, ChatGPT-4, and Copilot. While ChatGPT-4 and DeepSeek-v3 performed significantly better in the "postoperative" domain by providing "objectively true" answers ( P =0.038), Gemini and Copilot generated a greater proportion of "selected facts." Domain-specific variations were statistically significant only for Gemini ( P <0.001).

The results indicate that the reliability of AI-assisted language models in delivering medical information is subject to variation depending on the specific topic addressed. In its first comparative assessment within this study, DeepSeek-v3 outperformed the other evaluated models in terms of informational accuracy.