评估自然语言处理工具在标记胸部X-ray报告中的性能和偏差
Samantha M Santomartino1, John R Zech1, Kent Hall1
1From Drexel University College of Medicine, Philadelphia, Pa (S.M.S.); Department of Radiology, Columbia University Irving Medical Center, New York, NY (J.R.Z.); Department of Radiology, Wake Forest University Health Sciences Center, Winston-Salem, NC (K.H.); Department of Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, Baltimore, Md (J.J., V.P.); and Department of Diagnostic Imaging, St. Jude Children's Research Hospital, 262 Danny Thomas Plc, Memphis, TN 38105-3678 (P.H.Y.).
放射学报告注释的自然语言处理 (NLP) 工具总体上是准确的,但显示出显著的人口偏差,特别是在老年患者中. 需要进一步的研究来提高人工智能的公平性.
科学领域:
- 医疗信息学 医疗信息学
- 放射学中的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 自然语言处理 (NLP) 被广泛用于注释放射学数据集,以训练深度学习 (DL) 模型.
- 这些NLP方法的准确性和潜在的人口偏差,特别是在不同的患者群体中,需要进行彻底的调查.
- 了解这些局限性对于开发医疗成像中公平的AI至关重要.
研究的目的:
- 为了评估四个NLP放射学报告标签工具的准确性.
- 在不同患者群体中评估这些NLP工具中的人口偏差.
- 为了比较两个不同的胸部X-ray数据集的性能.
主要方法:
- 追溯性研究分析了MIMIC (n=692) 和IU (n=3665) 数据集 (2022年4月至2024年4月) 的胸部X光学报告.
- 四个NLP工具 (CheXpert,RadReportAnnotator,GPT-4,cTAKES) 对14个胸部疾病标签的放射科医生注释进行了评估.
- 绩效指标包括准确性和错误率;偏差被评估使用皮尔森基平方测试跨人口亚组.
主要成果:
- 这四种NLP工具都在两个数据集上显示出高的整体准确性.
- 对于大多数工具,在不同年龄组中观察到错误率的显著差异 (P < .001).
- 对于CheXpert,RRA和cTAKES的老年患者 (例如80岁以上) 的表现较差,对于GPT-4的60-80岁年龄组,表现较差.
结论:
- 虽然用于放射学报告注释的NLP工具总体上是准确的,但存在显著的人口偏差.
- 老年患者群体不成比例地受到较差的NLP性能的影响.
- 解决这些偏见对于在放射学中负责任地实施AI至关重要.
更多相关视频
相关概念视频
Radiological Investigation I: X-ray and CT
Assessment of Airway, Skin Color, and Use of Accessory Muscles
Introduction
The initial evaluation of a patient's respiratory system...
Radiological Investigation III: Pulmonary Angiogram and PET Scan
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Assessment of Respiration
Subjective Assessment: Nurses interview the patient to gather information directly during the subjective assessment. It includes questions about the individual's medical history, medications, and symptoms, focusing on past respiratory conditions like...
Radiological Investigation II: MRI and Ventilation Perfusion Scan
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
Respiratory System Abnormal Finding I: Inspection and Percussion
Inspection Findings
During an inspection, several findings may suggest the presence of respiratory distress or disease. Pursed-lip breathing, where exhalation is slowed by...


