Performance of AI chatbots in answering questions on tooth bleaching: a multilingual comparative study

Area of Science:

• Artificial Intelligence in Dentistry
• Restorative Dentistry Applications
• AI Chatbot Performance Evaluation

Background:

• AI chatbots are increasingly used for health information, but their reliability in specialized fields like restorative dentistry is under-evaluated.
• This study assesses the accuracy and consistency of AI chatbot responses concerning dental bleaching.

Purpose of the Study:

• To compare the accuracy and consistency of five AI chatbot systems in providing information on dental bleaching.
• To evaluate the performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, Gemini, and Claude-Instant in restorative dentistry.

Main Methods:

• Fifteen dental bleaching questions (undergraduate and specialist levels) were posed to five AI chatbots in Turkish and English.
• Responses were evaluated by two experts on a five-point Likert scale, with data analyzed using three-way ANOVA.
• Inter-rater agreement was assessed using Cohen's kappa coefficient.

Main Results:

• Chatbot type significantly impacted response accuracy (p < 0.001), with ChatGPT-4 demonstrating the highest accuracy.
• Claude-Instant and GPT-3.5 also performed well, while Gemini and Bing Chat showed significantly lower accuracy (p < 0.001).
• No significant main effects were found for question language or knowledge level, but significant interactions between chatbot type and these factors were observed.

Conclusions:

• The accuracy of AI chatbot information on dental bleaching is highly dependent on the specific AI model.
• Advanced models like ChatGPT-4 provide more accurate responses, highlighting the need for cautious interpretation and verification with professional guidance.
• AI chatbot outputs should not substitute professional clinical consultation for dental health information.