评估聊天GPT在紧急医疗委员会考试中的表现 问题:观察性研究
Mila Pastrak1, Sten Kajitani1, Anthony James Goodings1
1School of Medicine, University College Cork, Cork, Ireland.
JMIR AI
|July 4, 2025
概括
与ChatGPT-3.5.5相比,ChatGPT-4显示了紧急医疗委员会考试准备的精度提高. 使用闪存卡定制ChatGPT-4并没有显著提高性能,这表明其默认版本对医学教育有效.
科学领域:
- 医疗教育中的人工智能
- 医疗保健的自然语言处理.
- 医疗检查准备工具 医疗检查准备工具
背景情况:
- 人工智能 (AI) 平台如ChatGPT在医学的日益集成,需要了解它们在专业培训中的作用.
- 在医疗委员会考试准备中,特别是紧急医疗中,ChatGPT的有效性需要进行彻底的评估.
- 目前的研究还不清楚定制的人工智能模型是否为高风险的医疗评估提供了优越的性能,而不是默认版本.
研究的目的:
- 为了评估一个定制修改的ChatGPT-4的性能,增强了紧急医疗委员会准备材料,与其默认版本和ChatGPT-3.5.5的性能相比.
- 评估ChatGPT-4在回答板式紧急医疗问题的准确性.
- 确定ChatGPT是否适用于为准备标准化考试的医学学生和实习生提供辅助工具.
主要方法:
- 三个ChatGPT版本的比较分析:默认的ChatGPT-4,自定义的ChatGPT-4 (Anki闪卡甲板) 和ChatGPT-3.5.5.
- 从Rosh培训考试问题库中使用了598个随机选择的问题,用于紧急医疗委员会式的评估.
- 分析了每个ChatGPT版本的准确性,响应长度,跨医学子学科的性能和错误类型.
主要成果:
- 两种版本的ChatGPT-4在准确性方面都显著超过了ChatGPT-3.5 (P<.001).
- 默认的ChatGPT-4产生的响应时间远远长于自定义版本 (P<.001).
- 在默认和定制的ChatGPT-4版本 (P=.61) 之间没有观察到精度的显著差异,也没有在医学子学科之间观察到. 两种ChatGPT-4版本都预测了99%的通过率,而ChatGPT-3.5.5版本的通过率为85%.
结论:
- 新版本的ChatGPT,特别是ChatGPT-4,在准备紧急医疗委员会考试时表现出更好的性能.
- 调整ChatGPT-4以特定的准备材料,如Anki闪卡,并没有显著改善问题准确性.
- 默认版本的ChatGPT-4是一个有能力的医疗教育工具,提供准确的支持紧急医疗主题没有定制.
相关概念视频
Endoscopic Procedures V: ERCP
962
Endoscopic Retrograde Cholangiopancreatography (ERCP) is a diagnostic procedure that combines endoscopy and fluoroscopy to diagnose and treat conditions related to the bile ducts, pancreatic ducts, and gallbladder. This procedure is beneficial for identifying and addressing blockages, gallstones, strictures, and tumors within the biliary or pancreatic systems. ERCP is both diagnostic and therapeutic, offering the ability to visualize and treat identified problems in one session.
Patient...
Patient...
962
Cardiopulmonary Resuscitation III: AED Use
73
Introduction to AEDAn Automated External Defibrillator (AED) is a portable medical device that analyzes the heart's rhythm and, if necessary, delivers an electrical shock to help the heart re-establish an effective rhythm during sudden cardiac arrest (SCA). SCA occurs when the heart suddenly and unexpectedly stops beating, leading to a loss of blood flow to the brain and other vital organs. In such emergencies, time is of the essence, and using an AED, combined with Cardiopulmonary...
73
Pulmonary Embolism II: Diagnostic Studies and Interprofessional Care
46
Diagnosing Pulmonary EmbolismDiagnosing pulmonary embolism (PE) involves clinical assessment and advanced imaging tests. The preferred diagnostic tool is the spiral (helical) CT scan or CT angiography (CTA), which uses intravenous contrast media to visualize the pulmonary vasculature and identify emboli.A ventilation-perfusion (V/Q) scan is an alternative for patients unable to receive contrast media. This scan includes both perfusion and ventilation scanning. Perfusion scanning involves...
46
SBAR II: Application of SBAR
4.8K
SBAR is an effective communication tool used by healthcare professionals to communicate patient information accurately. SBAR stands for Situation, Background, Assessment, and Recommendation. For a better understanding, an example is given below.
SBAR Report from a Nurse to a Health Care Provider
S: "Hello, Dr. Smith. This is Jane, RN, from the Med Surg unit. I am calling to tell you about Ms. White in Room 210, who is experiencing increased pain and redness at her incision site. Her recent...
SBAR Report from a Nurse to a Health Care Provider
S: "Hello, Dr. Smith. This is Jane, RN, from the Med Surg unit. I am calling to tell you about Ms. White in Room 210, who is experiencing increased pain and redness at her incision site. Her recent...
4.8K


