Comparative Performance of Chatbots in Endodontic Clinical Decision Support: A 4-Day Accuracy and Consistency Study

Area of Science:

• Artificial Intelligence in Healthcare
• Dental Informatics
• Clinical Decision Support Systems

Background:

• Artificial intelligence (AI) is increasingly integrated into healthcare, raising concerns about its reliability and accuracy.
• Evaluating AI performance in specialized medical fields like endodontics is crucial for safe implementation.

Purpose of the Study:

• To assess the accuracy and consistency of five AI chatbots in answering endodontic questions.
• To compare performance across different difficulty levels (Basic, Intermediate, Advanced) and over time.

Main Methods:

• Five AI chatbots (ChatGPT-3.5, ChatGPT-4.o, Gemini 2.0 Flash, Copilot, Copilot Pro) were evaluated.
• A set of 60 clinically relevant endodontic questions, categorized by difficulty, were used.
• Questions were posed to chatbots over four days, with multiple daily intervals.

Main Results:

• ChatGPT-4.o achieved the highest overall accuracy (82.5%), excelling in Basic-level questions (95.0%).
• Copilot Pro exhibited the lowest accuracy (74.03%), with declining performance over days.
• Gemini's accuracy improved significantly over time, particularly in Advanced-level questions.

Conclusions:

• ChatGPT-4.o shows potential as a reliable AI tool for clinical decision-making in endodontics.
• Copilot and Copilot Pro demonstrated insufficient accuracy for clinical use.
• ChatGPT-3.5 and Gemini may be suitable for general queries but require careful consideration for complex cases.