利用大型语言模型来评估基于能力的医学教育中外科住院医生的叙事反质量
Benjamin Y M Kwan1, Zier Zhou2, Nick Rogoza3
1From the Department of Diagnostic Radiology, Queen's University, Kingston, Ontario, Canada.
概括
大型语言模型 (LLM) 在评估可信任专业活动 (EPA) 的外科住院反质量方面表现有前途. 精心调整的GPT-3.5实现了高准确度,这表明在基于能力的医学教育中自动化反评估的潜力.
科学领域:
- 医学教育 医学教育
- 人工智能的人工智能
- 进行外科手术培训.
背景情况:
- 基于能力的医学教育 (CBME) 增加了外科住院医生的叙事反量.
- 手动评估反质量是耗时的.
- 大型语言模型 (LLM) 提供了自动反评估的潜力.
研究的目的:
- 评估LLM在评估可信任专业活动 (EPA) 叙事反质量的表现.
- 使用学习质量评估 (QuAL) 框架将LLM生成的分数与人类评分进行比较.
- 探索提示技术和微调对LLM准确性的影响.
主要方法:
- 从外科住院EPA评估 (2017-2022) 中分析了2229个未被识别的叙事反评论.
- 使用生成预训练变压器 (GPT) -3.5和GPT-4模型进行反评估.
- 将LLM分数与人类分配的QuAL分数进行比较,使用F1分数作为主要指标.
主要成果:
- GPT-4在建议 (F1=0.901) 和连接 (F1=0.882) 中表现出色,但在证据 (F1=0.554) 中扎.
- 精心调整的GPT-3.5在所有QuAL维度中表现出卓越的性能:证据 (F1=0.827),建议 (F1=0.949) 和连接 (F1=0.933).
- 根据提示策略和模型架构,LLM的表现有所不同.
结论:
- 微调的GPT-3.5显示了在外科住院中自动化叙事反质量评估的巨大潜力.
- 法律学的有效性取决于任务,并受到与模型架构对齐的影响.
- 自动反评估可以支持基于能力的医学教育的持续质量改进.
相关概念视频
02:07Improving Translational Accuracy
3.5K
3.5K
02:07Improving Translational Accuracy
14.0K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
14.0K
01:15Nursing Evaluation
4.1K
The evaluation stage signals the end of the nursing process. The nurse gathers evaluative data to assess whether or not the patient has attained the expected results. Whereas the nurse collects data in the nursing assessment to identify the patient's health concerns, the evaluation stage data determines if the indicated health issues are resolved. Evaluative data collection includes two sections: the data acquired to evaluate patient outcomes and the time criteria for data collection.
4.1K
01:14SBAR II: Application of SBAR
5.6K
SBAR is an effective communication tool used by healthcare professionals to communicate patient information accurately. SBAR stands for Situation, Background, Assessment, and Recommendation. For a better understanding, an example is given below.
SBAR Report from a Nurse to a Health Care Provider
S: "Hello, Dr. Smith. This is Jane, RN, from the Med Surg unit. I am calling to tell you about Ms. White in Room 210, who is experiencing increased pain and redness at her incision site. Her recent...
SBAR Report from a Nurse to a Health Care Provider
S: "Hello, Dr. Smith. This is Jane, RN, from the Med Surg unit. I am calling to tell you about Ms. White in Room 210, who is experiencing increased pain and redness at her incision site. Her recent...
5.6K
01:23Methods of Documentation V: CBE
1.4K
Charting by Exception, or CBE, is a method of documentation used in healthcare, particularly in nursing, that focuses on documenting only significant or abnormal findings rather than recording every detail. This approach aims to streamline the documentation process, improve efficiency, and ensure that healthcare providers can quickly identify deviations from normalcy in patient assessments.
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
In CBE, healthcare professionals establish predefined standards of practice that define what constitutes...
1.4K