Response Analysis of Health Care Information Technologists in Japan Using Chat Generative Pretrained Transformer

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Health Information Technology

Background:

• The increasing integration of artificial intelligence (AI) like ChatGPT into healthcare necessitates evaluating its accuracy and reliability.
• Assessing AI's performance in non-English medical contexts is crucial for global adoption and safe implementation.
• Health care information technologist exams provide a standardized measure of knowledge in specific domains.

Purpose of the Study:

• To evaluate the reliability of the default ChatGPT model for assessing medical information knowledge.
• To determine the accuracy of ChatGPT's responses on health care, information technology, and health information systems exams in Japan.
• To identify specific areas where ChatGPT's medical knowledge is lacking, particularly concerning standards and laws.

Main Methods:

• Utilized the default model of ChatGPT for analysis.
• Administered a set of questions based on health care information technologist exams in Japan.
• Calculated average correct response rates across different subject fields.

Main Results:

• Achieved average correct response rates of 86% in health care, 93% in information technology, and 81% in health information systems.
• Demonstrated lower accuracy rates for questions pertaining to specific medical standards and laws.
• Identified a knowledge gap in regulatory and legal aspects within the medical domain.

Conclusions:

• ChatGPT exhibits a high degree of accuracy in general medical and IT knowledge for non-English contexts.
• The model's performance is significantly impacted by its limited understanding of specific medical standards and laws.
• Caution is advised when utilizing ChatGPT for medical information, as explanations may contain inaccuracies, especially regarding regulations.