GPT-4o and OpenAI o1 Performance on the 2024 Spanish Competitive Medical Specialty Access Examination:...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Natural Language Processing

Background:

• Generative artificial intelligence (AI) and large language models (LLMs) are rapidly advancing.
• LLMs show significant potential to transform medical education.
• Previous studies have evaluated chatbot performance on medical examinations.

Purpose of the Study:

• To assess the performance of GPT-4o and OpenAI o1 on the Médico Interno Residente (MIR) 2024 examination.
• To compare LLM performance against medical specialists and average candidates on a national medical test.
• To analyze LLM performance across various medical subjects and question difficulties.

Main Methods:

• 176 questions from the MIR 2024 examination were analyzed.
• Each question was presented individually to GPT-4o and OpenAI o1 to ensure independence.
• Accuracy was measured against official answers, with response consistency verified.
• Performance was benchmarked against a consensus of medical specialists and average MIR candidates.

Main Results:

• GPT-4o achieved 89.8% accuracy (90% with verification), and OpenAI o1 achieved 92.6% (93.2% with verification).
• Both LLMs and medical specialists significantly outperformed the average MIR candidate (56.6%).
• LLM accuracy decreased with increasing question difficulty and was slightly higher for clinical cases and positive questions.

Conclusions:

• GPT-4o and OpenAI o1 exhibit excellent and consistent accuracy on the MIR 2024 examination.
• LLMs offer promising opportunities for medical education and preparation for licensing exams.
• Future research should investigate factors influencing LLM accuracy and evaluate emerging AI models.