Pilot Study on Using Large Language Models for Educational Resource Development in Japanese Radiological Technologist...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Radiology Education

Background:

• Developing effective educational resources for medical licensure exams in non-English-speaking regions presents unique challenges.
• Large Language Models (LLMs) offer potential solutions for content generation and knowledge dissemination.

Purpose of the Study:

• To explore the application of LLMs in creating educational materials for the Japanese Radiological Technologist National Exam.
• To assess the performance of LLMs in generating explanations for different question types (image-based, calculation, textual).

Main Methods:

• Multiple-choice questions from the exam were categorized into image-based, calculation, and textual types.
• Explanatory texts were generated using Copilot, an LLM.
• The quality of generated explanations was assessed on a 0-4 point scale.

Main Results:

• LLMs demonstrated high performance in generating explanations for textual questions, showcasing strong capability with specialized content.
• Challenges were identified in generating accurate formulas and performing calculations for calculation questions.
• Interpreting complex medical images for image-based questions also posed difficulties for the LLMs.

Conclusions:

• LLMs have significant potential to enhance medical education resource development in non-English-speaking contexts.
• Addressing challenges requires integrating LLMs with programming capabilities for calculations and using precise prompts for image analysis.
• Educator involvement in validating LLM outputs and providing guidance is crucial for ensuring accuracy and effective utilization.