测试ChatGPT的能力和局限性,从转录的放射性发现中生成差异诊断
Shawn H Sun1, Kenneth Huynh1, Gillean Cortes1
1From the Department of Radiological Sciences, UCI Medical Center, University of California, Irvine, 101 The City Dr S, Orange, CA 92868 (S.H.S., K.H., G.C., R. Hill, J.T., R. Houshyar, V.Y., M.T.); Anaheim, Calif (L.Y.); and Division of Research, Kaiser Permanente of Northern California, Pleasanton, Calif (A.L.N.).
Radiology
|October 15, 2024
概括
聊天GPT (GPT-4) 在放射性差异诊断中显示出比GPT-3.5更高的准确性,但这两种模型都显示出诊断准确性的可靠性和可重复性问题.
科学领域:
- 人工智能在医学中的应用
- 放射学和医学成像学 医学成像学
- 自然语言处理自然语言处理.
背景情况:
- 对ChatGPT的医疗应用日益增长的兴趣需要严格的评估.
- 评估像ChatGPT这样的AI工具对于安全有效的临床整合至关重要.
- 了解AI在诊断过程中的局限性至关重要.
研究的目的:
- 从放射学发现中评估ChatGPT差异诊断的准确性,可靠性和可重复性.
- 为了比较GPT-3.5和GPT-4算法在生成差异诊断方面的性能.
- 识别AI驱动的诊断建议中的局限性,如幻觉和可重复性问题.
主要方法:
- 从教科书系列中使用了339个放射病例,涵盖了各种模式和病理.
- 在2023年4月3日至6月1日期间,标准化提示被输入了ChatGPT (GPT-3.5和GPT-4).
- 对准确性,可靠性 (幻觉) 和可重复性 (测试-重新测试) 的答案与教科书基础真相进行了比较.
主要成果:
- 对于最终诊断 (P <.001),GPT-4的整体准确性比GPT-3.5 (53.7%) 高 (66.1%).
- 比起GPT-3.5 (分别为39.9%和16.2%;P<.001),GPT-4产生的伪造引用 (14.3%) 和虚假陈述 (4.7%) 较少.
- 重复性差异很大,对顶级诊断的平均对对一致性在不同模型中从23%到49%不等.
结论:
- 与GPT-3.5相比,ChatGPT (GPT-4) 在放射差异诊断方面显示出更好的准确性和可靠性.
- 提示一个单一的诊断和使用GPT-4模型产生了最好的结果.
- 尽管有所改善,但重复性和幻觉仍然存在重大问题,需要进一步调查临床使用.
相关概念视频
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
86
Radiological investigations are paramount in the diagnosis and management of various pulmonary diseases. Two essential investigations are the Pulmonary Angiogram and the Positron Emission Tomography (PET) Scan.
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
86
01:30Radiological Investigation I: X-ray and CT
215
Radiological investigations, including X-rays and computed tomography (CT) scans, are critical for diagnosing and evaluating various medical conditions. These imaging techniques provide valuable insights into the body's internal structures, aiding in the detection of abnormalities, assessment of disease progression, and development of treatment strategies. This article delves into two primary radiological investigations, chest X-rays and CT scans, outlining their purpose, procedures, and...
215
01:10Computed Tomography
4.3K
Tomography refers to imaging by sections. Computed tomography (CT) is a non-invasive imaging technique that uses computers to analyze several cross-sectional X-rays to reveal minute details about structures in the body.
The technique was invented in the 1970s and is based on the principle that as X-rays pass through the body, they are absorbed or reflected at different levels. In the technique, a patient lies on a motorized platform while a computerized axial tomography (CAT) scanner rotates...
The technique was invented in the 1970s and is based on the principle that as X-rays pass through the body, they are absorbed or reflected at different levels. In the technique, a patient lies on a motorized platform while a computerized axial tomography (CAT) scanner rotates...
4.3K
01:29Positron Emission Tomography
4.0K
Positron emission tomography (PET) is a medical imaging technique involving radiopharmaceuticals — substances that emit short-lived radiation. Although the first PET scanner was introduced in 1961, it took 15 more years before radiopharmaceuticals were combined with the technique and revolutionized its potential.
One of the main requirements of a PET scan is a positron-emitting radioisotope, which is produced in a cyclotron and then attached to a substance used by the part of the body...
One of the main requirements of a PET scan is a positron-emitting radioisotope, which is produced in a cyclotron and then attached to a substance used by the part of the body...
4.0K
01:20Imaging Studies for Cardiovascular System III: X-Ray
149
The most common cardiovascular diagnostic test is an X-ray. It produces images of the heart, blood vessels, and adjacent structures.
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
149
01:30Radiological Investigation II: MRI and Ventilation Perfusion Scan
103
Description
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
103