Benchmarking of Large Language Models for the Dental Admission Test

Area of Science:

• Artificial Intelligence in Education
• Medical Education Technology
• Cognitive Science

Background:

• Large language models (LLMs) show promise in educational settings.
• Their performance on high-stakes admissions tests like the Dental Admission Test (DAT) is not well-understood.
• Assessing LLM capabilities is crucial for evaluating their role in test preparation.

Purpose of the Study:

• To evaluate the performance of 16 diverse large language models on a sample Dental Admission Test (DAT).
• To identify strengths and weaknesses of various LLMs in different DAT sections.
• To understand the challenges LLMs face in processing complex cognitive tasks relevant to dental admissions.

Main Methods:

• Evaluated 16 LLMs, including general-purpose, domain-specific, and open-source models.
• Conducted quantitative analysis of model accuracy across DAT sections.
• Performed qualitative thematic analysis by subject matter experts to identify specific challenges.

Main Results:

• GPT-4o and GPT-o1 excelled in text-based knowledge and comprehension sections (Natural Sciences, Reading Comprehension).
• Open-source models like Llama3-70B showed competitive performance in Reading Comprehension.
• All models struggled significantly with Perceptual Ability (image-based tasks), and fine-tuned medical models underperformed in critical thinking.

Conclusions:

• LLMs can aid in reinforcing factual knowledge for prospective dental students.
• Limitations in higher-order cognitive tasks and image-based reasoning necessitate guided integration with human instruction.
• Future innovations are needed to enhance LLM performance across all DAT cognitive skills.