在临床瘤学LLMs的比较评估
Nicholas R Rydzewski1, Deepak Dinakaran1,2,3, Shuang G Zhao4,5
1Radiation Oncology Branch, National Cancer Institute, National Institutes of Health, Bethesda, MD.
NEJM AI
|August 12, 2024
概括
大型语言模型 (LLM) 在瘤学中表现有前途,但存在显著的错误率. GPT-4表现最好,尽管持续评估对于AI在癌症护理中的安全临床整合至关重要.
科学领域:
- 在瘤学瘤学.
- 人工智能的人工智能
- 医疗信息学 医疗信息学
背景情况:
- 人工智能 (AI) 和大型语言模型 (LLM) 为医学信息检索提供了潜力.
- 瘤学,其复杂的数据和指导方针,可以显著受益于LLM的进步.
- 对LLM进行基准评估对于安全的临床瘤学应用至关重要.
研究的目的:
- 为了全面评估在瘤学LLMs.
- 确定医疗专业人员信任LLM响应的策略.
主要方法:
- 在2044个瘤学问题上测试了五个LLM (LLaMA 1,PaLM 2,Claude-v1,GPT-3.5,GPT-4).
- 通过快速重复和自我评估的信心分数来评估输出一致性.
- 验证了50个瘤学问题的新一套表现,以避免训练数据重叠.
主要成果:
- 在模型中观察到显著的性能异质性 (ANOVA,P<0.001).
- GPT-4的表现优于其他模型,超过了人类50百分点的基准值.
- 性能因学科不同而有所不同,在临床瘤学子领域和女性占主导地位的恶性瘤中精度较低.
结论:
- 在接受测试的LLM中,GPT-4在瘤学问题上表现最高.
- 所有的LLM都表现出临床上显著的错误率,包括过度自信和不准确.
- 持续的标准化评估对于引导AI工具在临床瘤学中安全整合至关重要.
相关概念视频
Cancer Survival Analysis
334
Cancer survival analysis focuses on quantifying and interpreting the time from a key starting point, such as diagnosis or the initiation of treatment, to a specific endpoint, such as remission or death. This analysis provides critical insights into treatment effectiveness and factors that influence patient outcomes, helping to shape clinical decisions and guide prognostic evaluations. A cornerstone of oncology research, survival analysis tackles the challenges of skewed, non-normally...
334
Comparing the Survival Analysis of Two or More Groups
166
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...
166
Combination Therapies and Personalized Medicine
4.9K
Combining two or more treatment methods increases the life span of cancer patients while reducing damage to vital organs or tissue from the overuse of a single treatment. Combination therapy also targets different cancer-inducing pathways, thus reducing the chances of developing resistance to treatment.
The combination of the drug acetazolamide and sulforaphane is a good example of combination therapy to treat cancer. The cells in the interior of a large tumor often die due to the hypoxic and...
The combination of the drug acetazolamide and sulforaphane is a good example of combination therapy to treat cancer. The cells in the interior of a large tumor often die due to the hypoxic and...
4.9K


