肺栓塞成像的放射性决策:大型语言模型的准确性和可靠性-Bing,Claude,ChatGPT和困惑
Pradosh Kumar Sarangi1, Suvrankar Datta2, M Sarthak Swarup3
1Department of Radiodiagnosis, All India Institute of Medical Sciences Deoghar, Deoghar, Jharkhand, India.
The Indian journal of radiology & imaging
|September 25, 2024
概括
人工智能 (AI) 聊天机器人在支持肺栓塞 (PE) 的成像决策方面表现出不同的准确性. 困惑在开放式问题中脱而出,而Bing在选择所有适用的问题中领先,这表明在临床集成之前需要人工智能改进.
科学领域:
- 放射学和医学成像学 医学成像学
- 医疗保健中的人工智能
- 临床决策支持系统 临床决策支持系统
背景情况:
- 人工智能聊天机器人有可能改善临床决策和简化医疗保健工作流程.
- 然而,它们在特定临床场景 (如肺栓塞) 的放射性决策支持中的作用需要进一步研究.
- 本研究评估了著名的大型语言模型 (LLM) 对PE成像建议的准确性.
研究的目的:
- 评估四个领先的LLM (Microsoft Bing,Claude,ChatGPT 3.5,Perplexity) 在提供疑似肺栓塞 (PE) 初始成像临床决策支持方面的准确性和可靠性.
- 为了比较不同问题格式的LLM的表现 (开放式与选择所有适用的).
- 评估放射科医生对LLM反应的协议的一致性.
主要方法:
- 根据美国放射学学院的适当性标准,开发了四种PE病例情景的变体.
- 三名放射科医生向LLM提出了开放式 (OE) 和选择所有适用的 (SATA) 问题.
- 通过使用类内相关系数 (ICC) 来评分,规范化和分析答案的准确性和评审者之间的可靠性.
主要成果:
- 困惑在OE问题上表现出最高准确度 (0.83),而Claude的准确度最低 (0.58).
- 对于SATA问题,Bing的准确度最高 (0.96),困惑度最低 (0.56).
- 总的来说,OE问题 (0.73) 的得分高于SATA问题 (0.68),放射科医生对SATA的认同很强 (ICC=0.875),但对OE的认同很差 (ICC=-0.067).
结论:
- 在疑似PE的临床决策支持中,LLM准确度存在显著差异.
- 困惑和Bing在不同的问题格式 (分别是OE和SATA) 中表现出强势.
- 目前的LLM不一致性需要进一步的微调和放射科医生的仔细选择,以确保可靠的临床整合.
相关概念视频
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
88
Radiological investigations are paramount in the diagnosis and management of various pulmonary diseases. Two essential investigations are the Pulmonary Angiogram and the Positron Emission Tomography (PET) Scan.
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
88
01:30Radiological Investigation II: MRI and Ventilation Perfusion Scan
105
Description
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
105
01:20Imaging Studies for Cardiovascular System III: X-Ray
150
The most common cardiovascular diagnostic test is an X-ray. It produces images of the heart, blood vessels, and adjacent structures.
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
Definition and Purpose
An X-ray, or radiograph, is a non-invasive method that uses ionizing radiation to take images of internal structures. It is mainly used in cardiac imaging to examine the heart, lungs, and major blood vessels, aiming to identify abnormalities in the heart's size, shape, and position, such as heart failure, congenital defects, and vascular...
150