Performance of Enhanced Large Language Models on Prosthodontic Multiple-Choice Questions

Area of Science:

• Artificial Intelligence in Dentistry
• Natural Language Processing for Dental Education
• Large Language Model Performance Evaluation

Background:

• Large language models (LLMs) are increasingly explored for specialized domains like dentistry.
• Enhancement strategies such as retrieval-augmented generation (RAG), in-context learning (ICL), and majority voting aim to improve LLM accuracy and reliability.
• Evaluating LLM performance on domain-specific questions is crucial for their practical application.

Purpose of the Study:

• To assess the effectiveness of retrieval-augmented generation (RAG), in-context learning (ICL), and majority voting in enhancing large language models (LLMs) for prosthodontic questions.
• To compare the performance of enhanced LLMs against their base versions using both Chinese and English prosthodontic question sets.
• To analyze error types and identify areas where LLM enhancements yield significant improvements.

Main Methods:

• Two base large language models (OpenAI o1, DeepSeek-R1) were enhanced using RAG, ICL, and majority voting techniques.
• Standardized Chinese-language multiple-choice questions (C-MCQs) and English-language multiple-choice questions (E-MCQs) in prosthodontics were used for evaluation.
• Performance was measured by answer correctness, error analysis, and statistical comparison (χ² tests) between base and enhanced models.

Main Results:

• Enhanced LLMs demonstrated statistically significant accuracy improvements on C-MCQs (P < .001) and a reduction in knowledge-based errors (P < .008).
• While enhanced LLMs showed higher accuracy on E-MCQs, the difference was not statistically significant (P = .145).
• The benefits of LLM enhancement, particularly in accuracy and error reduction, were primarily observed in the Chinese-language question set.

Conclusions:

• Inference-time enhancement strategies (RAG, ICL, majority voting) significantly boost LLM accuracy for Chinese prosthodontic questions.
• These enhancements effectively mitigate certain errors but do not consistently yield statistically significant improvements for English questions.
• Enhanced LLMs show promise for processing dental-related tasks and hold potential for applications in dentistry education.