Potential of Large Language Models in Generating Multiple-Choice Questions for the Japanese National Licensure...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Allied Health Professions Education

Background:

• Developing high-quality assessment tools is crucial for professional licensure exams.
• Large language models (LLMs) offer potential for automating content generation in education.
• Evaluating LLM performance in non-English contexts for specialized exams is an emerging area.

Purpose of the Study:

• To assess the efficacy of a customized large language model (LLM) in generating multiple-choice questions (MCQs) for the Japanese National Licensure Examination for Physical Therapists.
• To evaluate the quality of LLM-generated MCQs and their explanations in a non-English language (Japanese).

Main Methods:

• A customized ChatGPT model, "Physio Exam GPT," was trained on 340 MCQs from previous Japanese Physical Therapist licensure exams.
• Prompts and outputs were conducted in Japanese, covering general and practical physical therapy topics.
• Generated MCQs and explanations were evaluated by two independent reviewers using a 10-point Likert scale across five criteria.

Main Results:

• The LLM achieved 100% accuracy in generating both general and practical MCQs.
• Average reviewer scores ranged from 7.0 to 9.8 for general questions and 6.7 to 9.8 for practical questions.
• Overall, the generated MCQs and explanations were rated favorably, indicating high quality.

Conclusions:

• Large language models (LLMs) demonstrate significant potential for efficiently generating high-quality MCQs in non-English languages like Japanese.
• LLMs can adapt to diverse linguistic settings, reduce educator workload, and enhance educational assessment resources.
• This study provides a foundation for applying LLMs in educational settings across non-English-speaking regions for professional licensure exams.