Performance of AI chatbots in answering questions on tooth bleaching: a multilingual comparative study

Artificial intelligence (AI) chatbots are increasingly used in healthcare to provide health-related information and answer patient questions. However, their reliability in specialized dental fields such as restorative dentistry remains insufficiently evaluated. This study aimed to evaluate and compare the accuracy and consistency of responses generated by five artificial intelligence chatbot systems-ChatGPT-3.5 and ChatGPT-4 (OpenAI), Bing Chat (Microsoft), Gemini (Google), and Claude-Instant (Anthropic)-regarding dental bleaching.

Fifteen frequently asked questions about dental bleaching, identified by restorative dentistry specialists, were categorized as undergraduate-or specialist-level questions. All questions were submitted to five artificial intelligence chatbots in both Turkish and English. Each question was asked three times per day over three consecutive days using standardized prompts. Responses were independently evaluated by two experts using a five-point Likert scale, and mean scores were calculated. A three-way ANOVA was conducted to assess the effects of chatbot type, knowledge level, and question language on response accuracy. Inter-rater agreement between evaluators was assessed using Cohen's kappa coefficient. Statistical significance was set at p < 0.05.

Chatbot type had a significant effect on response accuracy (p < 0.001, η² = 0.405). ChatGPT-4 showed the highest accuracy, followed by Claude-Instant and GPT-3.5, whereas Gemini and especially Bing Chat demonstrated significantly lower performance (p < 0.001). Question language and knowledge level showed no significant main effects (p > 0.05). Significant interactions were observed between chatbot type and knowledge level and between chatbot type and language (p < 0.001). No significant differences were observed across days or time periods (p > 0.05).

The accuracy of chatbot-generated information regarding dental bleaching depends strongly on the specific AI model used. Advanced large language models, particularly ChatGPT-4, generate more accurate and consistent responses than other evaluated systems. AI chatbots should therefore not be considered interchangeable sources of clinical information, and their outputs should be interpreted cautiously and verified with professional guidance.

These findings highlight the importance of critically evaluating AI-generated health information and emphasize that chatbot responses should not replace professional clinical consultation.