人工智能能像教授一样评分吗? 对比人工智能和医学系学生短答案临床推理考试的教师得分
Arvind Rajan1, Seth McKenzie Alexander2,3, Christina L Shenvi4
1Department of Medicine, The University of North Carolina School of Medicine, Chapel Hill, NC, USA.
Advances in health sciences education : theory and practice
|August 6, 2025
概括
OpenAI的GPT-4o大型语言模型 (LLM) 可以有效地对医学教育中的叙事短答题 (SAQ) 进行评分,在许多问题类型中与教师评分器相等. 虽然对效率有希望,但人工智能分级应该补充,而不是取代人类评估.
科学领域:
- 医学教育 医学教育
- 人工智能在评估中的作用
- 自然语言处理自然语言处理.
背景情况:
- 医学院通常依赖于多选题 (MCQs) 进行临床前评估,因为等级效率.
- 在评估高阶思维和学生推理过程方面,MCQ存在局限性.
- 像简短答案问题 (SAQ) 这样的替代评估更全面,但需要大量资源来评分.
研究的目的:
- 评估OpenAI的GPT-4o大型语言模型 (LLM) 在评分叙事短答题 (SAQ) 的有效性.
- 在基于案例的学习 (CBL) 考试中,将LLM评分表现与人类教师评分器进行比较.
- 评估LLM分级精度,Bloom分类层次的性能,以及与问题复杂性的相关性.
主要方法:
- 一项涉及1450名学生在CBL考试中对SAQ的反应的研究.
- 通过GPT-4o LLM分配的分数与经验丰富的教师评级者的分数进行比较.
- 统计分析包括启动可信区间以确定分级等价性 (在±5%的范围内).
- 二次分析包括精度的类内相关系数 (ICC) 和问题复杂性的回归分析.
主要成果:
- GPT-4o LLM评分显示了与教师评分的总体等价性 (平均差异: -0.55%,95% CI: -1.53%至+0.45%).
- 在Bloom分类的记忆,应用和分析级别下分类的SAQ中证实了等价性.
- 人工智能分类表现出高精度 (ICC = 0.993).
- 在更复杂的问题上,LLM和教师之间的分数差异较大 (R2 = 0.6199,p < 0.0001),特别是在理解和评估水平上.
结论:
- 基于LLM的评分,就像GPT-4o一样,可以作为医学教育中SAQ初步评分的宝贵工具.
- 人工智能分级有可能提高本科医学教育考试的效率和质量.
- 这些发现强调了教师参与的重要性,表明人工智能工具应该增强,而不是取代人类评级者,特别是在更高层次的认知技能方面.
相关概念视频
Critical Thinking II
3.1K
Critical thinking is a cognitive process with several attributes. The attributes of critical thinking include the following:
3.1K
Reliability and Validity
13.2K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.2K
Patient-centered Care
2.3K
Patient-centered care involves delivering care beyond inpatient hospitalization. Reflective practice can enhance a patient-centered approach. Reflective practice is a process of reasoning that considers all aspects of the present situation, including practicalities, learning from personal practice, and consideration of patient needs. Patients appreciate care decisions made while considering their input. Involving the patient in their care provides the patient with a sense of contribution rather...
2.3K
Critical Thinking I
3.6K
Critical thinking helps decision-making and allows nurses to recognize barriers to success and find solutions to possible issues. It helps to brainstorm and implement ideas to achieve goals. Critical thinking helps acknowledge and state workflow inefficiencies while improving management techniques. Nurses understand the value of critical thinking and look for fellow nurses with critical thinking skills to upgrade their professional standards. Critical thinking can advance a nurse's career...
3.6K
Methods of Documentation III: PIE
1.5K
Problem-intervention-evaluation (PIE) is a systematic approach to documentation used in healthcare settings for clinical decision-making and patient care planning. It is a structured approach to organizing patient data based on problems, interventions, and evaluations. Here's a breakdown of its key features and considerations:
1.5K
Nursing Evaluation
3.6K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
3.6K


