Related Experiment Video
Updated: May 28, 2026
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
Benchmarking Multi-Modal Artificial Intelligence Models Against Student Performance: The Role of Question
1Department of Orthodontics and Paediatric Dentistry, College of Dentistry, Qassim University, Al-Mulaydah, Qassim, Saudi Arabia.
Background:
Objective Structured Practical Dental Examinations (OSPDEs) are high-stakes assessments in dental education. This study compared the performance of multimodal Artificial Intelligence (AI) models in a preclinical Orthodontics and Paediatric Dentistry OSPDE with retrospective student cohort performance and examined how question characteristics impact performance.
Methods:
The performance of Claude-3.5-Sonnet, Gemini-1.5-Pro, GPT-4o, and 48 students was compared using 80 OSPDE questions evenly divided between visually supported multiple-choice (MCQ) and fill-in-the-blank (FB) formats. Questions were categorised by visual type, visual dependency, course domain, Bloom's taxonomy cognitive levels, stem length, and difficulty. Accuracy and agreement rates were analysed using Chi-Square tests and ANOVA (p < 0.05).
Results:
In MCQ formats, AI models performed comparably to students, with students achieving slightly higher accuracy (83.3%). Students outperformed AI on difficult questions, while AI excelled on easier ones. GPT-4o and Gemini-1.5-Pro showed higher agreement rates (95%) than Claude-3.5-Sonnet (79.4%). In FB formats, students significantly outperformed Claude-3.5-Sonnet (69.5% vs. 30%), with only students and GPT-4o meeting passing standards. Students excelled in table-based and combination questions, where AI struggled (near 0% accuracy). Both GPT-4o and students performed well on visually dependent questions (p < 0.001). Students outperformed AI in "remember," "apply," and "analyse" tasks, while GPT-4o excelled in "understand." Both performed better on shorter stems. Across domains, students and GPT-4o outperformed Claude-3.5-Sonnet in Orthodontics.
Conclusion:
Multimodal AI models demonstrated comparable performance to students in MCQ formats but showed significant limitations in FB formats, though methodological differences limit direct comparisons. GPT-4o demonstrated the most promising performance among AI models, but improvements are needed in orthodontics, interpreting tables and complex visuals, handling difficult questions, and performing cognitive tasks such as "apply" and "analyse".
