GPT-4/4V's performance on the Japanese National Medical Licensing Examination

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine

Background:

• Artificial Intelligence (AI) is rapidly advancing, impacting various professional fields, including medicine.
• AI's clinical capabilities are increasingly evaluated through standardized medical examinations.
• This study focuses on assessing a specific AI, ChatGPT, against the Japanese National Medical Licensing Examination (NMLE).

Purpose of the Study:

• To evaluate the performance of OpenAI's ChatGPT on the Japanese National Medical Licensing Examination (NMLE).
• To assess AI's ability to answer NMLE questions, including those with visual components (images, illustrations, pictures).

Main Methods:

• Acquired six years of NMLE questions (112th-117th) from the Japanese Ministry of Health, Labour and Welfare.
• Converted NMLE questions into JavaScript Object Notation (JSON) format.
• Developed an API utilizing GPT-4 and GPT-4V(ision) to process and answer questions, including image-based queries.

Main Results:

• Image-based questions constituted 30.1% (723/2400) of the NMLE over six years.
• GPT-4/4V consistently exceeded the minimum passing score across all examined years.
• Correct answer rates were 73.5% for basic medical knowledge, 74.7% for clinical knowledge, and 68.7% for image-based questions.

Conclusions:

• AI, specifically GPT-4/4V, met minimum performance criteria on the NMLE, irrespective of question format (text, image, illustration).
• The findings suggest significant potential for AI applications in medical education and assessment.