评估放射学大语言模型 系统审查 标题和摘要 选
Siddhant Dogra1, Soroush Arabshahi2, Jason Wei1
1New York University Langone Health. Department of Radiology. New York, NY (S.D., J.W., E.H.).
Academic radiology
|August 23, 2025
概括
大型语言模型 (LLM) 对放射学系统审查查具有前景,但需要谨慎的及时设计和人类监督以实现强大的性能. 他们的决策行为和信心校准需要进一步评估.
科学领域:
- 医疗研究中的人工智能
- 放射学系统审查
- 自然语言处理应用
背景情况:
- 系统性审查对于放射学中的证据综合至关重要.
- 通过大型语言模型 (LLM) 自动化标题和摘要选可以提高效率.
- 在这种情况下,评估LLM的绩效,稳定性和决策是必不可少的.
研究的目的:
- 评估各种LLM用于放射学系统审查查的性能,稳定性和决策.
- 调查快速框架和信心校准对LLM准确性的影响.
- 在遇到分歧时评估LLM的稳定性,并探索自主搜索功能.
主要方法:
- 在两个放射学系统审查中对五个LLM (GPT-4o,GPT-4o mini,Gemini 1.5 Pro,Gemini 2.0 Flash,Llama 3.3 70B) 进行了比较.
- 任务包括二元/三元分类,信任评分和分歧的重新分类.
- 使用OpenAI和Gemini深度研究工具进行了自主PubMed检索试点.
主要成果:
- LLM 具有高特异性,但敏感性可变,F1 评分在 0. 389 至 0. 854 之间.
- 三级分类的结果是低的禁用率 (<5%) 和适度的敏感度增长.
- 模型表现出权威偏见, 在分歧中偏好人类标签, 尽管GPT-4o表现出更大的抵抗力.
结论:
- 在放射学中,LLM显示出系统审查查的潜力.
- 精心及时的工程和人为循环监督对于可靠的LLM应用至关重要.
- 需要进一步的研究来优化LLM的整合和解决决策偏见.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
575
07:15Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model
Published on: August 16, 2020
6.9K
相关概念视频
01:13Radiological Investigation III: Pulmonary Angiogram and PET Scan
167
Radiological investigations are paramount in the diagnosis and management of various pulmonary diseases. Two essential investigations are the Pulmonary Angiogram and the Positron Emission Tomography (PET) Scan.
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
Pulmonary Angiogram
A Pulmonary Angiogram is an invasive procedure involving injecting a contrast medium through a catheter threaded into the pulmonary artery or the right side of the heart to visualize the pulmonary vasculature. Computed Tomography (CT) scans have mainly replaced this...
167
01:30Radiological Investigation II: MRI and Ventilation Perfusion Scan
221
Description
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
Magnetic Resonance Imaging (MRI) and Ventilation Perfusion Scans are two radiological investigations that offer detailed diagnostic images of the body, particularly lung structures.
MRI
MRI uses magnetic fields and radiofrequency signals to distinguish between normal and abnormal tissues. This technology provides a more detailed diagnostic image than CT scans, enabling it to characterize pulmonary nodules, stage bronchogenic carcinoma, and evaluate inflammatory activity in...
221
01:30Radiological Investigation I: X-ray and CT
414
Radiological investigations, including X-rays and computed tomography (CT) scans, are critical for diagnosing and evaluating various medical conditions. These imaging techniques provide valuable insights into the body's internal structures, aiding in the detection of abnormalities, assessment of disease progression, and development of treatment strategies. This article delves into two primary radiological investigations, chest X-rays and CT scans, outlining their purpose, procedures, and...
414