在使用Murad工具评估案例报告和案例系列的方法质量时,人类和GPT-4之间的一致性
Zin Tarakji1,2, Adel Kanaan1,2, Samer Saadi1,2
1Evidence-based Practice Center, Kern Center for the Science of Healthcare Delivery, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA.
BMC medical research methodology
|November 4, 2024
概括
GPT-4显示了与评估案例报告方法质量的人类审稿人的中度一致,使用Murad工具进行评估. 虽然有希望,但人工智能援助需要人类监督,以进行系统的审查.
科学领域:
- 医学研究方法论医学研究方法论.
- 医疗保健中的人工智能
背景情况:
- 评估案例报告和系列的方法质量是困难的,因为人类判断的可变性和时间限制.
- 在质量评估中,评估人类审稿人和人工智能 (GPT-4) 之间的协议至关重要.
研究的目的:
- 通过使用Murad评估工具评估案例报告和系列的方法质量,评估人类审查员和GPT-4之间的协议.
主要方法:
- 在Scopus中搜索了引用穆拉德评估工具 (2023-2024) 的系统评论.
- 开发了一个GPT-4代理来使用该工具的8个信号问题来评估方法质量.
- 人类审查者与GPT-4之间的估计观察到的一致性和一致系数.
主要成果:
- 包括797个案例报告/系列;观察到的一致性在41.91%至80.93%之间.
- 最低的共识是关于案件选择的问题.
- 除了案例选择问题之外,GPT-4显示出与自身的高度一致.
结论:
- GPT-4显示了与人类审稿人对案例报告/系列的方法质量评估的中度协议.
- 目前的GPT-4性能很有希望,但单独进行系统性审查是不够的;需要人类-AI合作.
相关概念视频
Multiple Comparison Tests
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
Methods of Documentation II: POMR
The Problem-Oriented Medical Record (POMR) revolutionized medical record-keeping by introducing a systematic approach focusing on the patient's problems rather than merely listing symptoms. Dr. Lawrence Weed's introduction of this method in the 1960s marked a significant advancement in medical documentation. The POMR framework consists of four key components: the database, problem list, plan of care, and progress notes.


