Performance of Large Language Models in the Japanese Public Health Nurse National Examination: Comparative...

Area of Science:

• Artificial Intelligence
• Medical Education
• Public Health Nursing

Background:

• Large language models (LLMs) show promise in medical and nursing exams.
• No prior studies assessed LLM performance on the Japanese Public Health Nurse National Examination.
• This exam requires specialized community and public health nursing knowledge.

Purpose of the Study:

• To compare the performance of multiple LLMs on the Japanese Public Health Nurse National Examination.
• To evaluate the utility of LLMs in public health nursing education.

Main Methods:

• Three LLMs (GPT-4o, Claude Opus 4, Gemini 2.5 Pro) were tested.
• All 110 questions from the 111th examination were used with standardized prompts.
• Accuracy rates were calculated and compared statistically, with questions categorized by format, content, and selection type.

Main Results:

• All LLMs surpassed the 60% passing criterion.
• Accuracy rates ranged from 85.5% (GPT-4o) to 92.7% (Gemini 2.5 Pro), with no significant differences among models.
• All models performed worse on multiple-choice questions compared to single-choice questions.

Conclusions:

• LLMs demonstrate high potential for public health nursing examinations.
• Limitations exist in complex reasoning, particularly with multiple-choice questions.
• LLMs can serve as educational support tools, but cautious implementation is advised in specialized nursing education.