Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis

Area of Science:

• Artificial Intelligence in Education
• Dental Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly used in healthcare and dental education.
• The accuracy of LLMs in specialized dental examination tasks is not well-established.

Purpose of the Study:

• To systematically review and meta-analyze the performance of LLMs in answering dental examination questions.
• To assess the pooled accuracy and identify factors influencing LLM performance in this domain.

Main Methods:

• Systematic review and meta-analysis of studies from PubMed, Embase, Scopus, and Web of Science.
• Extracted data on question types, LLM versions, and accuracy rates.
• Estimated pooled accuracy using a random-effects model and assessed heterogeneity and bias.

Main Results:

• Included 39 studies, with ChatGPT-4 most frequently evaluated.
• Pooled LLM accuracy was 63.7% (95% CI: 60.3%-67.1%) with high heterogeneity.
• ChatGPT-4 and Copilot showed the highest pooled accuracies (approx. 73% and 75%), with ChatGPT-4 outperforming earlier versions.

Conclusions:

• LLMs exhibit moderate accuracy for dental exam questions, insufficient for autonomous clinical decisions.
• LLMs can be valuable adjuncts in dental education and exam preparation when limitations are acknowledged.
• Further research into structured prompting and retrieval-augmented approaches is warranted.