Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in...

Area of Science:

• Artificial Intelligence in Healthcare
• Dental Informatics
• Clinical Decision Support Systems

Background:

• Generative artificial intelligence large language models (LLMs) are increasingly used across disciplines, including dentistry.
• Concerns exist regarding the accuracy and reliability of LLM-generated content in clinical settings.

Purpose of the Study:

• To comparatively evaluate the accuracy of four leading large language models (LLMs) in dentistry.
• Assessing Bard, ChatGPT-3.5, ChatGPT-4, and Bing Chat responses to clinical dental questions.

Main Methods:

• LLMs were queried with 20 clinically relevant, open-ended dentistry questions.
• Answers were scored by experienced faculty against established scientific evidence using a rubric.
• Statistical analysis (Friedman and Wilcoxon tests) compared LLM performance.

Main Results:

• ChatGPT-4 statistically outperformed other LLMs (ChatGPT-3.5, Bing Chat, Bard).
• All evaluated LLMs demonstrated occasional inaccuracies, generality, outdated content, and lack of references.
• Qualitative assessment revealed irrelevant, vague, or partially inaccurate information from LLMs.

Conclusions:

• LLMs show potential for evidence-based dentistry but require judicious use due to current limitations.
• Critical thinking and subject matter expertise remain paramount for dental practitioners.
• Further research, validation, and regulatory oversight are essential for safe LLM integration into dental practice.