Evaluating the Evidence Base of Large Language Models in Answering Clinical Questions Related to Denture Care and...

Large language models (LLMs) have gained significant attention and are increasingly considered as decision-support tools in healthcare. Nevertheless, their accuracy in relation to established prosthodontic guidelines remains underexplored. The purpose of this study was to evaluate and compare the evidence-based potential of answers provided by 4 LLMs to common clinical questions regarding denture care and maintenance.

A total of 10 open-ended questions pertinent to denture care and maintenance were posed to 4 distinct LLMs, namely ChatGPT 4o, Google Gemini Advanced, Microsoft Copilot, and DeepSeek V3. The answers were evaluated independently by 2 prosthodontists against established guidelines for comprehensiveness, scientific accuracy, clarity, and relevance. Differences were analyzed using Friedman and Wilcoxon signed-rank tests. To assess intra-evaluator reliability, a reevaluation of the LLM responses was performed after 4 weeks, and Cronbach's α and interclass correlation coefficient (ICC) were used (α=.05).

ChatGPT 4o and Google Gemini Advanced recorded the highest mean scores (8.5 out of 10), followed by DeepSeek V3 (8.4 out of 10) and Microsoft Copilot (8.0 out of 10). No statistically significant differences were observed among the models.

In this limited set of denture-care questions, LLMs often provided high-quality responses that aligned with ACP denture care guidelines, although occasional inaccuracies were observed. Their use shows potential as additional decision-support tools, but insights are limited to routine denture hygiene and maintenance questions. Caution and expert supervision are still crucial, as LLMs can't replace dental professionals in prosthodontic treatment or patient care.