Performance Comparison of a Domain-Specific Chatbot and General-Purpose Chatbots in Dental Traumatology

Area of Science:

• Dentistry
• Artificial Intelligence
• Medical Informatics

Background:

• Artificial intelligence (AI) chatbots are increasingly used in dental traumatology.
• Concerns about the reliability of AI in dental trauma care persist.
• This study assesses the accuracy of the Dental Trauma Evo AI chatbot for dental fractures and luxations.

Purpose of the Study:

• To evaluate the accuracy of the Dental Trauma Evo AI chatbot.
• To compare its performance against other leading AI chatbots in dental traumatology.
• To assess chatbot responses to various question formats (MCQs, True/False, Yes/No).

Main Methods:

• 45 validated questions on dental fractures and luxations were used.
• Four chatbots (Dental Trauma Evo, ChatGPT-4o, DeepSeek R1, Google Gemini 2.5) were tested.
• Answers were verified for accuracy and consistency using Fisher's exact test.

Main Results:

• Dental Trauma Evo achieved the highest overall accuracy (85.43%).
• Google Gemini, DeepSeek, and ChatGPT-4o followed with accuracies of 81.72%, 80.24%, and 79.75%, respectively.
• Chatbot performance varied by question type, with no statistically significant difference observed.

Conclusions:

• The Dental Trauma Evo chatbot demonstrates promising preliminary performance based on International Association of Dental Traumatology guidelines.
• Further research and clinical validation are essential for practical implementation.
• Enhancements to the AI model are recommended for optimal clinical utility.