Leading large language models on a periodontology knowledge test.

Area of Science:

• Artificial Intelligence
• Dental Research
• Periodontology

Background:

• Large language models (LLMs) are increasingly integrated into clinical and educational settings.
• Limited data exists on LLM performance within specialized dental fields.
• Assessing LLMs' capabilities in periodontology is crucial for understanding their potential applications.

Purpose of the Study:

• To evaluate the performance of four large language models (LLMs) in periodontology.
• To compare general-purpose and research-focused LLMs on a validated question set.
• To determine the impact of role-specific priming on LLM accuracy.

Main Methods:

• Four LLMs (ChatGPT-4o, DeepSeek-R1, Consensus, Perplexity) were tested on 50 multiple-choice periodontology questions.
• Each LLM underwent five independent trials under primed and non-primed conditions.
• Statistical analyses included ANOVA and t-tests, with item-level analysis for difficult questions.

Main Results:

• Overall accuracy across all LLMs was 65.0%, with no significant differences between models.
• Role-specific priming did not enhance LLM performance.
• Four questions were never answered correctly, and many others showed low accuracy, particularly those requiring detailed procedural knowledge or rare facts.

Conclusions:

• Current LLMs possess moderate domain knowledge in periodontology.
• LLMs do not yet meet the reliability standards for unsupervised clinical decision support in this field.
• Further development is necessary to improve LLM accuracy and applicability in specialized dental practice.