评估ChatGPT对Bloom分类学的掌握使用精神病学医学考试问题:混合方法研究
Anne Herrmann-Werner1,2, Teresa Festl-Wietek1, Friederike Holderried1,3
1Tübingen Institute for Medical Education, Faculty of Medicine, University of Tübingen, Tübingen, Germany.
像GPT-4这样的大型语言模型在医学考试中表现出高的成功率,但在较低级别的认知任务中却很困难. 记忆事实和理解概念的错误凸显了人工智能的局限性.
科学领域:
- 人工智能在医学中的应用
- 医疗教育 技术 技术 医学教育
背景情况:
- 像GPT-4这样的大型语言模型 (LLM) 在医学教育中越来越多地被采用.
- 法律学士容易产生事实上的不准确性,称为"幻觉".
- 在LLM错误和Bloom的分类学认知水平之间的关系还不清楚.
研究的目的:
- 为了评估GPT-4在精神病学医学考试问题上的表现.
- 通过布鲁姆的分类学来分析GPT-4的错误.
主要方法:
- 利用了307个从医学院考试中选择题的精神病学医学数据集.
- 用两个快速版本向GPT-4提出问题:详细和简短.
- 进行定量和定性分析,根据Bloom的分类法对错误进行分类.
主要成果:
- GPT-4的成功率很高:93% (详细提示) 和91% (简短提示).
- 正确回答的问题比错误回答的问题要困难得多 (P<.001).
- 错误主要发生在"记忆" (29/68) 和"理解" (23/68) 认知层面,涉及事实回忆和概念理解.
结论:
- 在医学多选题检查中,GPT-4表现出高准确度.
- 通过布鲁姆的分类学进行的分析揭示了事实回忆,逻辑推理和概念应用中的错误.
- 模型偏差和概率输出生成可能会导致这些"幻觉".
更多相关视频
06:28E-Patient Counseling Trial E-PACO: Computer Based Education versus Nurse Counseling for Patients to Prepare for Colonoscopy
Published on: August 1, 2019
00:08A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
相关概念视频
Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
Assessment of the Cardiovascular System I: Subjective Data
Initial Enquiry
Ask the patient about their primary concern and thoroughly explore all reported symptoms.
Medical History
Investigate past illnesses affecting the cardiovascular system, such as angina, anemia, rheumatic fever, congenital heart disease, stroke, thrombophlebitis, dysrhythmias, varicosities
Inquire about symptoms...
Data Collection III
The principles to begin the physical assessment include conducting a comprehensive or problem-related history in a quiet, well-lit room, emphasizing privacy and comfort for the...
Chronic Obstructive Pulmonary Disease-IV: Assessement and Diagnostic Studies
Medical History
The Scientific Method
Assessment of the Gastrointestinal System I: Subjective Data
Health History
The initial step in assessing the GI system is obtaining a comprehensive health history. This includes inquiring about the patient's history or presence of problems...
