Analysis of multimodal large language models on visually-based questions in the Japanese National Examination for...

The importance of oral health is globally recognized, which has increased the demand for qualified dental hygienists. This study assessed the performance of multimodal large language models (LLMs), on the Japanese National Examination for Dental Hygienists, focusing on their ability to answer visually-based questions and evaluating image-recognition capabilities.

The 34th Japanese National Examination for Dental Hygienists (March 2025) supplied 213 multiple-choice questions (74 text-only, 139 visually-based). Five multimodal LLMs were tested: OpenAI o3-mini-high (o3-mh), ChatGPT-4.5 Preview (GPT-4.5), Gemini 2.0 Flash Thinking Experimental (Gemini 2.0), Gemini 2.5 Pro Experimental (Gemini 2.5), and Claude 3.7 Sonnet (Claude 3.7). Performance was evaluated by comparing LLM answers to official correct answers. Cochran's Q test and McNemar's tests with Bonferroni correction were used for statistical analysis.

Gemini 2.5 achieved the highest overall correct response rate (85.0 %), followed by Claude 3.7 (77.5 %), o3-mh (77.0 %), GPT-4.5 (76.1 %), and Gemini 2.0 (75.1 %). For text-only questions, Claude 3.7 (91.9 %) performed best. On visually-based questions, Gemini 2.5 was superior (82.0 %), while other models scored around 70-73 %. Gemini 2.5 significantly outperformed, GPT-4.5 and Gemini 2.0 overall, and GPT-4.5 and Claude 3.7 on visually-based questions.

Multimodal LLMs, particularly Gemini 2.5, demonstrate significant proficiency on the Japanese National Examination for Dental Hygienists, including questions with visual elements. These findings suggest a growing potential for LLMs as educational tools in dental hygiene. However, current limitations in accuracy and reliability necessitate further refinement and cautious integration into educational and clinical settings.