Response Analysis of Health Care Information Technologists in Japan Using Chat Generative Pretrained Transformer

This study used the default model of ChatGPT on health care information technologist exams in Japan to evaluate the reliability of this model for medical information knowledge in non-English languages. The average correct response rates for all questions in the health care, information technology, and health information systems fields were 86%, 93% and 81%, respectively. ChatGPT lacked knowledge of standards and laws; therefore, the accuracy rates for related questions were low. Thus, the study concludes that careful attention is required when using ChatGPT, because several of the explanations it provides lack correct descriptions.