五个人工智能模型在USMLE第一步问题上的表现:一项比较观测研究
Dania El Natour1, Mohamad Abou Alfa1, Ahmad Chaaban1
1Faculty of Medicine, Beirut Arab University, Hariri Bldg, 4th Fl., Beirut, LB.
JMIR AI
|February 9, 2026
概括
新的人工智能 (AI) 模型在美国医学执照考试 (USMLE) 第一步问题上进行了评估. 格罗克获得了最高的准确性 (91.6%),超过了ChatGPT-4和其他AI工具,特别是在基于图像的问题上.
科学领域:
- 医学教育 医学教育
- 人工智能在医学中的应用
- 医学执照考试 医学执照考试
背景情况:
- 人工智能 (AI) 模型越来越多地被整合到医学教育中.
- 以前像ChatGPT这样的AI模型在USMLE样式的问题上表现出熟练.
- 新兴的人工智能工具需要对医疗领域的准确性和可靠性进行更新的比较评估.
研究的目的:
- 为了比较五个人工智能模型的性能:Grok,ChatGPT-4,Copilot,Gemini和DeepSeek.
- 评估USMLE第一步免费120个问题集的AI准确性和一致性.
- 分析不同医学科目和问题格式 (基于文本,基于图像,基于案例) 的表现.
主要方法:
- 从2025年2月10日至3月5日进行的横截面观测研究.
- 在每个AI模型中,使用标准化提示符,对每个AI模型进行了119个USMLE样式问题.
- 模型回答每一个问题三次以评估一致性;问题按类型分类;统计分析包括千平方和费舍尔的精确测试.
主要成果:
- 格洛克获得了最高的整体得分 (91.6%),其次是Copilot (84.9%),Gemini (84.0%),ChatGPT-4 (79.8%) 和DeepSeek (72.3%).
- 深度搜索在以图像为基础的问题上遇到了困难 (0%的准确性),但在仅以文本为基础的场景 (89.6%) 中表现出色.
- 格罗克在基于图像 (91.3%) 和基于案例 (89.7%) 的问题上表现优异,一致性高 (100%).
结论:
- 人工智能模型在医疗领域的表现各不相同,Grok表现出最高的准确性和一致性.
- 像Grok和Copilot这样的新型号可以与像ChatGPT-4这样的既定工具竞争.
- 由于人工智能工具的快速发展和不断变化的能力,对人工智能工具的持续评估至关重要.
相关概念视频
01:11Observational Studies
11.1K
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One...
11.1K
01:14Steps in the Modeling Process
687
Albert Bandura's theory of observational learning identifies four critical processes: attention, retention, motor reproduction, and reinforcement or motivation.
Attention is the first necessary component for observational learning. It involves focusing on what the model is doing and saying. For example, if you decide to take a drawing class to enhance your skills, you need to pay close attention to the instructor's words and hand movements. The characteristics of the model significantly...
Attention is the first necessary component for observational learning. It involves focusing on what the model is doing and saying. For example, if you decide to take a drawing class to enhance your skills, you need to pay close attention to the instructor's words and hand movements. The characteristics of the model significantly...
687
02:30Naturalistic Observations
17.5K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
17.5K
03:08Rate-Determining Steps
37.5K
Relating Reaction Mechanisms
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
In a multistep reaction mechanism, one of the elementary steps progresses significantly slower than the others. This slowest step is called the rate-limiting step (or rate-determining step). A reaction cannot proceed faster than its slowest step, and hence, the rate-determining step limits the overall reaction rate.
The concept of rate-determining step can be understood from the analogy of a 4-lane freeway with a short-stretch of traffic-bottleneck caused due to...
37.5K
02:26Comparing Copy Number Variations and SNPs
18.8K
Sequencing of the human genome has opened up several best-kept secrets of the genome. Scientists have identified thousands of genome variations that exist within a population. These variations can be a single nucleotide or a larger chromosomal variation.
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
Copy number variations or CNVs are the structural variations that cover more than 1kb of DNA sequence. The single nucleotide polymorphism (SNP), on the other hand, is a single nucleotide change or a point mutation that is found in more than 1%...
18.8K
01:12Observational Learning
1.0K
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
1.0K