What the performance of multimodal LLMs on a national licensing exam teaches us about occupational therapy education

Multimodal large language models (LLMs) are increasingly adopted in occupational therapy (OT) education, offering new opportunities for learning and assessment. However, their ability to handle professional licensing examinations remains unexplored. This study aimed to provide educational insights for OT educators by analyzing how multimodal LLMs perform on the Japanese National Examination for Occupational Therapists (JNEOT), particularly focused on visually-based questions.

The complete 199-item 2025 JNEOT (consisting of 177 text-only questions and 22 visually-based questions) was presented to OpenAI o3, Gemini 2.5, GPT-4.5, and Claude 3.7 in a zero-shot manner. Official answer keys served as the reference standard. Overall correct response rates and subscores were calculated for text-only versus visually-based and general versus practical items. We examined differences using Cochran's Q, and significant omnibus results were followed by Bonferroni-corrected McNemar tests.

All models achieved passing-level scores (82.4-91.0%) on text-based questions but showed a marked decline on visually based items (50.0-68.2%). No significant inter-model differences were found for visual tasks. Ten OT-specific questions were answered incorrectly by all models.

These findings suggest that LLMs can effectively reinforce foundational OT knowledge but remain limited in visual interpretation and contextual reasoning. For educators, the results offer practical guidance on using LLMs to enhance theoretical learning while maintaining OT educator-guided approaches for reflective and experiential competence development.

Not applicable.