聊天GPT-3.5和波兰胸部外科专业考试:一个绩效评估考试
Adam Mitręga1, Dominika Kaczyńska1, Mikołaj Magiera1
1Students' Scientific Association of Computer Analysis and Artificial Intelligence at the Department of Radiology and Nuclear Medicine, Medical University of Silesia, Katowice, Poland.
概括
在胸部外科专家考试中,ChatGPT-3.5得分为42.2%,低于60%的合格门. 这种人工智能工具在临床知识和一致性方面存在局限性,需要谨慎进行专业评估.
科学领域:
- 医学教育 医学教育
- 人工智能在医学中的应用
- 胸部外科手术 胸部外科手术
背景情况:
- 人工智能 (AI) 的快速发展为医疗应用提供了新的机会.
- 人工智能在医疗保健中的可靠性和局限性需要彻底评估.
- 评估AI在专业医疗检查中的表现至关重要.
研究的目的:
- 为了评估ChatGPT-3.5语言模型的有效性.
- 评估ChatGPT-3.5在胸部外科国家专家考试 (PES) 的表现.
主要方法:
- 分析了2015年PES考试中的120个测试题.
- 问题按主题,临床特征和认知要求分类.
- 聊天GPT-3.5回答了每一个问题五次;使用统计测试 (χ2,克鲁斯卡尔-瓦利斯,曼-惠特尼,斯皮尔曼) 和弗莱斯的k系数.
主要成果:
- 聊天GPT-3.5获得了42.2%的正确答案,低于60%的通过分数.
- 在临床和非临床问题之间发现了统计学上显著的差异 (p=0.041).
- 正确答案有较高的信心 (p<0.001),但信心与心理指标没有相关性;响应一致性中等 (k=0.341).
结论:
- 聊天GPT-3.5的表现相当于专家考试中的失败分数.
- 虽然响应信心与正确性相关,但AI在临床知识和一致性的局限性需要谨慎.
- 需要进一步的研究来了解AI在专业医学知识评估中的作用.
相关概念视频
01:25Pulmonary Function Tests
718
Pulmonary Function Tests (PFTs)
Pulmonary Function Tests are crucial diagnostic tools for assessing respiratory function, particularly in patients with chronic respiratory disorders. They comprehensively evaluate lung volumes, ventilatory function, breathing mechanics, diffusion, and gas exchange. These tests help diagnose pulmonary diseases and play a significant role in monitoring disease progression, evaluating disability, and assessing response to therapy.
PFTs involve using a spirometer, a...
Pulmonary Function Tests are crucial diagnostic tools for assessing respiratory function, particularly in patients with chronic respiratory disorders. They comprehensively evaluate lung volumes, ventilatory function, breathing mechanics, diffusion, and gas exchange. These tests help diagnose pulmonary diseases and play a significant role in monitoring disease progression, evaluating disability, and assessing response to therapy.
PFTs involve using a spirometer, a...
718
01:26Flail Chest-II
493
Managing flail chest, a condition characterized by a segment of the chest wall moving independently from the rest of the thoracic cage, requires a comprehensive approach. It includes a thorough assessment of the patient's condition, a diagnostic evaluation to determine the extent of the injury, and the implementation of appropriate medical interventions tailored to the individual's needs.
Assessment:
1. Clinical Evaluation:
History:
Assessment:
1. Clinical Evaluation:
History:
493
01:26Endoscopic Studies II: Thoracocentesis
1.2K
Thoracentesis(Thoracocentesis), commonly known as pleural tap, is a medical procedure where a 22 gauge needle is inserted into the pleural space, the area between the lung and chest wall. This procedure is commonly performed to diagnose or treat various respiratory disorders.
Description
Excess pleural fluid or air may accumulate in some respiratory disorders in the thoracic cavity. To treat pleural effusion, a physician conducts thoracentesis by carefully piercing the chest wall and entering...
Description
Excess pleural fluid or air may accumulate in some respiratory disorders in the thoracic cavity. To treat pleural effusion, a physician conducts thoracentesis by carefully piercing the chest wall and entering...
1.2K
01:24Physical Assessment of the Respiratory Tract II: Palpation
1.6K
Physical assessment of the respiratory tract is critical in identifying potential health issues. One key component of this assessment is palpation, a technique healthcare providers use to assess the body for abnormalities. This content explores the method of palpation in evaluating the respiratory tract, focusing on thoracic palpation and tactile fremitus.
Thoracic Palpation
Thoracic palpation detects tenderness, masses, lesions, respiratory excursions, and vocal fremitus. The nurse assesses...
Thoracic Palpation
Thoracic palpation detects tenderness, masses, lesions, respiratory excursions, and vocal fremitus. The nurse assesses...
1.6K
01:29Physical Assessment of the Respiratory Tract III: Percussion
2.7K
The respiratory system, fundamental to life, consists of complex structures responsible for gas exchange. The percussion assessment is critical to understanding this system's health and functionality. This non-invasive assessment technique allows healthcare providers to evaluate the density or aeration of the lungs, thereby identifying potential abnormalities.
Percussion in Respiratory Assessment
Percussion evaluates underlying tissue composition with audible and tactile vibrations,...
Percussion in Respiratory Assessment
Percussion evaluates underlying tissue composition with audible and tactile vibrations,...
2.7K
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
387
Radiological investigations are paramount in the diagnosis and management of various pulmonary diseases. Two essential investigations are the Pulmonary Angiogram and the Positron Emission Tomography (PET) Scan.
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
387