Related Experiment Video
Updated: Feb 11, 2026
04:35
Author Spotlight: Simulation and Analysis of the Temperature Rise of Ring Main Unit Equipment
Published on: July 5, 2024
2.4K
Performance of 5 AI Models on United States Medical Licensing Examination Step 1 Questions: Comparative Observational
Dania El Natour1, Mohamad Abou Alfa1, Ahmad Chaaban1
1Faculty of Medicine, Beirut Arab University, Beirut, Lebanon.
JMIR AI
|February 9, 2026
Summary
New artificial intelligence (AI) models were evaluated on United States Medical Licensing Examination (USMLE) Step 1 questions. Grok achieved the highest accuracy (91.6%), outperforming ChatGPT-4 and other AI tools, especially on image-based questions.
Area of Science:
- Medical Education
- Artificial Intelligence in Medicine
- Medical Licensing Examinations
Background:
- Artificial intelligence (AI) models are increasingly integrated into medical education.
- Previous AI models like ChatGPT showed proficiency in USMLE-style questions.
- Emerging AI tools require updated comparative evaluations for accuracy and reliability across medical domains.
Purpose of the Study:
- To compare the performance of five AI models: Grok, ChatGPT-4, Copilot, Gemini, and DeepSeek.
- To assess AI accuracy and consistency on the USMLE Step 1 Free 120-question set.
- To analyze performance across different medical subjects and question formats (text-based, image-based, case-based).
Main Methods:
- Cross-sectional observational study conducted from February 10 to March 5, 2025.
- 119 USMLE-style questions were administered to each AI model using standardized prompts.
- Models answered each question thrice to evaluate consistency; questions were categorized by type; statistical analysis included Chi-square and Fisher's exact tests.
Main Results:
- Grok achieved the highest overall score (91.6%), followed by Copilot (84.9%), Gemini (84.0%), ChatGPT-4 (79.8%), and DeepSeek (72.3%).
- DeepSeek struggled with image-based questions (0% accuracy) but excelled in text-only scenarios (89.6%).
- Grok demonstrated superior performance on image-based (91.3%) and case-based (89.7%) questions, with high consistency (100%).
Conclusions:
- AI models exhibit varied performance across medical domains, with Grok showing top accuracy and consistency.
- Newer models like Grok and Copilot are competitive with established tools like ChatGPT-4.
- Ongoing assessment of AI tools is crucial due to their rapid development and evolving capabilities.
Related Concept Videos
01:11Observational Studies
11.1K
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
11.1K
01:14Steps in the Modeling Process
687
Albert Bandura's theory of observational learning identifies four critical processes: attention, retention, motor reproduction, and reinforcement or motivation.
Attention is the first necessary component for observational learning. It involves focusing on what the model is doing and saying. For example, if you decide to take a drawing class to enhance your skills, you need to pay close attention to the instructor's words and hand movements. The characteristics of the model significantly...
Attention is the first necessary component for observational learning. It involves focusing on what the model is doing and saying. For example, if you decide to take a drawing class to enhance your skills, you need to pay close attention to the instructor's words and hand movements. The characteristics of the model significantly...
687
02:30Naturalistic Observations
17.5K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
17.5K
03:08Rate-Determining Steps
37.5K
Relating Reaction Mechanisms
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
37.5K
02:26Comparing Copy Number Variations and SNPs
18.8K
Sequencing of the human genome has opened up several best-kept secrets of the genome. Scientists have identified thousands of genome variations that exist within a population. These variations can be a single nucleotide or a larger chromosomal variation.
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
18.8K
01:12Observational Learning
1.0K
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
1.0K