在3000个USMLE®风格问题中分析影响ChatGPT性能的问题特征
Michael Alfertshofer1,2, Samuel Knoedler3,4, Cosima C Hoch5
1Department of Oral and Maxillofacial Surgery, Ludwig-Maximilians-University Munich, Munich, Germany.
Medical science educator
|March 27, 2025
概括
像ChatGPT这样的人工智能 (AI) 在医学教育中显示出潜力,但需要评估. 这项研究发现,ChatGPT正确回答了57.7%的USMLE Step 2CK问题,性能因医疗专业和问题难度而异.
科学领域:
- 医学教育 医学教育
- 人工智能的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 人工智能 (AI) 和大型语言模型 (LLM) 在医学上表现有前途.
- 对AI在医疗应用中的性能进行全面评估至关重要.
- 本研究专门评估了ChatGPT在USMLE Step 2CK问题上的表现.
研究的目的:
- 为了评估ChatGPT在USMLE Step 2CK练习问题上的准确性和性能.
- 分析不同医学专业,问题类型和难度等级的AI性能.
- 为开发AI耐药医学检查提供信息,并指导AI在医学教育中的整合.
主要方法:
- 从AMBOSS学习平台提取了3000个基于文本的USMLE Step 2CK练习问题.
- 手动输入问题到ChatGPT进行分析.
- 评估了ChatGPT在各种类别和难度级别的准确性和性能.
主要成果:
- 在测试问题上,ChatGPT的整体准确率为57.7%.
- 根据医疗类别的表现差异很大,最高分数在"男性生殖系统" (71.7%) 和最低的"免疫系统" (46.3%).
- 基于表格问题的表现较低,问题难度与准确性之间存在负相关性 (r=-0.285,p<0.001).
结论:
- 聊天GPT的表现接近USMLE Step 2CK的通过门,但根据类别,类型和难度而异.
- 这些发现有助于教育工作者创建AI耐药考试,并将AI工具集成到教学策略中.
- 了解人工智能的局限性是学生将人工智能作为积极的学习辅助工具而不是替代品的关键.
更多相关视频
10:17Improving Student Outcomes with an Adaptable Molecular Cloning Course-Based Undergraduate Research Experience
Published on: November 15, 2024
894
06:28E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
Published on: August 1, 2019
8.2K
相关概念视频
01:20Comparing the Survival Analysis of Two or More Groups
107
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...
107
02:16Surveys
14.7K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.7K
01:12Analysis of Population Pharmacokinetic Data
205
Analysis of population pharmacokinetic data involves studying the behavior of drugs within diverse populations to understand their pharmacokinetic parameters. Traditional pharmacokinetic methods typically involve collecting samples from a few individuals and estimating these parameters. While these methods are commonly used, they have limitations in capturing the variability in drug response among individuals or heterogeneous populations. Population pharmacokinetics is employed to address these...
205
01:17Cochran's Q Test
173
Cochran's Q Test is a nonparametric statistical test used to determine if there are potential differences in the outcomes of three or more related groups on a binary (yes/no) or dichotomous outcome. It is essentially an extension of the McNemar Test, which is limited to two related samples - Cochran's Q test can handle three or more related samples, making it more versatile in scenarios where subjects are measured under multiple conditions. The test statistic follows a Chi-Square...
173
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
59
Assessing the gastrointestinal (GI) system is a complex process that begins with collecting subjective data. This data, collected through patient interviews, provides crucial insights into the patient's health history, perception patterns, and lifestyle habits, all contributing significantly to GI health.
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
59
01:14Model Approaches for Pharmacokinetic Data: Compartment Models
64
Compartmental analysis is a widely adopted approach to characterizing drug pharmacokinetics. It uses compartment models that conceptualize the body as a collection of reversibly communicating compartments, each representing a group of tissues exhibiting similar drug distribution characteristics. The movement rate of the drug between these compartments is typically described by first-order kinetics.
Two primary types of compartment models are recognized: mammillary and catenary. The more...
Two primary types of compartment models are recognized: mammillary and catenary. The more...
64