ChatGPT (GPT-4V) Performance on the Healthcare Information Technologist Examination in Japan

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Medical Education Technology

Background:

• The rapid advancement of AI, particularly large language models like ChatGPT, presents new opportunities in healthcare informatics.
• The Healthcare Information Technologist (HCIT) certification in Japan validates professionals skilled in electronic health records (EHRs) and patient care improvement.
• Evaluating AI performance on professional certification exams is crucial for understanding its practical application in specialized fields.

Purpose of the Study:

• To assess the performance of ChatGPT (GPT-4V) on the Japanese Healthcare Information Technologist (HCIT) certification exam.
• To determine if ChatGPT (GPT-4V) meets the passing criteria for HCIT certification.
• To analyze ChatGPT (GPT-4V)'s accuracy across different question types, including those with and without images or calculations.

Main Methods:

• ChatGPT (GPT-4V) was administered 476 questions from the HCIT certification exams over three years.
• Performance was evaluated based on the overall correct answer rate.
• Accuracy was further analyzed by question category, format (simple-choice vs. multiple-choice), inclusion of images, and necessity of calculations.

Main Results:

• ChatGPT (GPT-4V) achieved a mean correct answer rate of 84%, surpassing the HCIT exam's passing threshold.
• Accuracy was higher for simple-choice questions compared to multiple-choice questions (P < 0.05).
• Performance was lower for questions involving images (P < 0.01) and calculations (P < 0.05) compared to text-only or non-calculation questions.

Conclusions:

• ChatGPT (GPT-4V) effectively meets the HCIT certification requirements, indicating a strong grasp of essential knowledge and skills.
• The AI's performance suggests it possesses the minimum competencies necessary for roles involving electronic health records.
• Further research can explore AI's role in healthcare education and professional development.