Large Language Models Evaluation of Medical Licensing Examination Using GPT-4.0, ERNIE Bot 4.0, and GPT-4o.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly explored for their potential in specialized fields.
• Evaluating LLM performance in high-stakes professional examinations is crucial for understanding their capabilities.
• The Chinese Medical Licensing Examination presents a rigorous test case for AI in healthcare education.

Purpose of the Study:

• To systematically evaluate and compare the performance of three advanced LLMs (GPT-4.0, ERNIE Bot 4.0, GPT-4o) on the 2023 Chinese Medical Licensing Examination.
• To analyze the accuracy of these LLMs across different domains of the medical licensing exam.
• To identify the strengths and limitations of each LLM for potential applications in medical training.

Main Methods:

• A dataset of 600 standardized questions from the 2023 Chinese Medical Licensing Examination was utilized.
• Performance analysis focused on three sections: Basic Medical Comprehensive, Clinical Medical Comprehensive, and Humanities and Preventive Medicine Comprehensive.
• Accuracy metrics were calculated for each LLM (GPT-4.0, ERNIE Bot 4.0, GPT-4o) against the examination's benchmark.

Main Results:

• Both ERNIE Bot 4.0 and GPT-4o demonstrated superior performance compared to GPT-4.0.
• ERNIE Bot 4.0 and GPT-4o achieved accuracy scores exceeding the national pass mark.
• GPT-4.0 showed lower performance in this specific medical licensing examination context.

Conclusions:

• Advanced LLMs like ERNIE Bot 4.0 and GPT-4o show significant potential for assisting in medical education and assessment.
• The findings highlight the need for further research into LLM capabilities and limitations in multilingual medical contexts.
• Integrating AI tools like LLMs could offer a pathway to enhance medical training and examination processes.