Evaluating AI Chatbot Performance Against Prosthodontic Residents on the National Prosthodontic Resident Examination...

Area of Science:

• Artificial Intelligence
• Dental Education
• Prosthodontics

Background:

• The integration of artificial intelligence (AI) into specialized fields like prosthodontics is rapidly advancing.
• Assessing AI performance against human experts is crucial for understanding its potential and limitations.

Purpose of the Study:

• To compare the performance of seven AI chatbots against U.S. prosthodontic residents on the National Prosthodontic Resident Examination (NPRE).
• To evaluate AI capabilities in a high-stakes academic setting within prosthodontics.

Main Methods:

• AI chatbots (ChatGPT 3.5, 4, 4o, Gemini, Claude 3, Copilot, Llama) responded to NPRE questions from 2011-2023.
• Responses were compared to official answer keys and resident performance.
• Statistical analysis included repeated measures ANOVA and Bonferroni post-hoc tests.

Main Results:

• Significant differences in performance were observed across AI models (p<0.001).
• ChatGPT 3.5 outperformed other AI models and matched resident performance.
• Copilot performed significantly worse than residents, while others showed comparable, yet variable, results.

Conclusions:

• AI chatbots, particularly ChatGPT 3.5, demonstrate proficiency on specialized prosthodontic exams.
• Variability in AI performance underscores the need for critical assessment before adoption in dental education.
• AI shows promise as a supplementary tool, but reliability concerns require further investigation.