Limitations of knowledge competency and error patterns in large language models based on orthodontic licensing...

Area of Science:

• Artificial Intelligence in Dentistry
• Medical Education Technology
• Orthodontic Assessment

Background:

• Large language models (LLMs) are increasingly explored for medical applications.
• Evaluating LLM performance in specialized fields like orthodontics is crucial for understanding their utility and limitations.
• Orthodontic licensing examinations provide a standardized benchmark for assessing medical knowledge and clinical reasoning.

Purpose of the Study:

• To assess the performance limitations of two leading LLMs, Deepseek-R1 (DS) and ChatGPT-4 (GPT), in the field of orthodontics.
• To compare LLM performance across different orthodontic knowledge domains and identify common error types.

Main Methods:

• A total of 396 text-based questions from the Chinese National Orthodontic Specialist Licensing Examination were used.
• Questions were categorized by knowledge domains (foundational biomechanics, cross-disciplinary integration, specialized theory, clinical decision-making) and error types (factual, logical, semantic).
• Performance metrics including accuracy and error rates were compared between DS and GPT.

Main Results:

• DS achieved significantly higher overall accuracy (80.3%) compared to GPT (52.3%) (p < 0.001).
• DS also demonstrated superior performance in foundational knowledge (79.8% vs 43.4%) and cross-disciplinary domains (81.0% vs 53.0%).
• Factual errors were most common for both models, while DS showed a higher rate of logical errors.

Conclusions:

• DS exhibits greater proficiency than GPT in orthodontic knowledge assessment via standardized exams.
• Both LLMs demonstrate limitations in specialized orthodontic domains requiring complex clinical reasoning.
• While DS shows promise for AI-assisted orthodontic training and evaluation, clinician oversight remains essential due to persistent errors and domain-specific weaknesses.