Performance assessment of artificial intelligence chatbots (ChatGPT-4 and Copilot) for sharing insights on 3D-printed...

To evaluate and compare the performance of OpenAI's ChatGPT-4 and Microsoft Copilot in providing information on 3D-printed orthodontic appliances, with a focus on the accuracy, completeness of the content, and response generation time.

This cross-sectional study proceeded in five stages. Initially, three orthodontists created a total of 125 questions concerning 3D printed orthodontic appliances of which 105 questions were finalized to be incorporated into the study by a panel of senior orthodontists. These questions were subsequently organized into 15 distinct domains. Both chatbots were presented with the questions under consistent conditions, using the same laptop and internet setup. A stopwatch was used to record response times. The responses were anonymized and evaluated by seven orthodontists with extensive experience, who scored accuracy and completeness based on standardized tools. Through discussion, evaluators reached a consensus on each score, ensuring reliability.

Spearman's correlation revealed a moderate to strong negative correlation between accuracy and completeness for both chatbots (p≤0.001). The negative correlation observed between accuracy and completeness scores, particularly prominent in Copilot, indicates a trade-off between these qualities in some responses. Mann-Whitney U tests confirmed significant differences in accuracy and completeness between the chatbots (p≤0.001), though response time differences were not statistically significant (p=0.204). Cohen's Kappa results implied little to no consistency between the two models on the assessed parameters (p>0.05).

ChatGPT-4 outperformed Microsoft Copilot in accuracy and completeness, providing more precise and comprehensive information on 3D-printed orthodontic appliances demonstrating a greater ability to handle complex, and detailed requests in this area.