Related Experiment Video
Updated: Jul 25, 2025

The 4 Mountains Test: A Short Test of Spatial Memory with High Sensitivity for the Diagnosis of Pre-dementia Alzheimer's Disease
Published on: October 13, 2016
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Soshi Takagi1, Takashi Watari1,2,3,4, Ayano Erabi1
1Faculty of Medicine, Shimane University, Izumo, Japan.
Background:
The competence of ChatGPT (Chat Generative Pre-Trained Transformer) in non-English languages is not well studied.
Objective:
This study compared the performances of GPT-3.5 (Generative Pre-trained Transformer) and GPT-4 on the Japanese Medical Licensing Examination (JMLE) to evaluate the reliability of these models for clinical reasoning and medical knowledge in non-English languages.
Methods:
This study used the default mode of ChatGPT, which is based on GPT-3.5; the GPT-4 model of ChatGPT Plus; and the 117th JMLE in 2023. A total of 254 questions were included in the final analysis, which were categorized into 3 types, namely general, clinical, and clinical sentence questions.
Results:
The results indicated that GPT-4 outperformed GPT-3.5 in terms of accuracy, particularly for general, clinical, and clinical sentence questions. GPT-4 also performed better on difficult questions and specific disease questions. Furthermore, GPT-4 achieved the passing criteria for the JMLE, indicating its reliability for clinical reasoning and medical knowledge in non-English languages.
Conclusions:
GPT-4 could become a valuable tool for medical education and clinical support in non-English-speaking regions, such as Japan.
Related Concept Videos
Comparing Experimental Results: Student's t-Test
Multiple Comparison Tests
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
Sensitivity, Specificity, and Predicted Value
Sensitivity is the...
Improving Translational Accuracy

