一个新的医学LLM评估基准,阐明临床领域的安全性和有效性
Shirui Wang1, Zhihui Tang2, Huaxia Yang3
1Medlinker Intelligent and Digital Technology Co. Ltd., Beijing, China.
NPJ digital medicine
|December 26, 2025
概括
评估临床使用的大型语言模型 (LLM) 需要一个强大的基准. 我们的研究引入了临床安全有效性双轨基准 (CSEDB),揭示了性能差距,特别是在高风险情况下.
科学领域:
- 人工智能在医学中的应用
- 临床信息学 临床信息学
- 医疗决策支持系统 医疗决策支持系统
背景情况:
- 大型语言模型 (LLM) 显示出临床决策支持的潜力.
- 在评估它们在医疗保健中的安全性和有效性方面存在重大挑战.
- 标准化的评估框架对于可靠的临床部署至关重要.
研究的目的:
- 引入临床安全有效性双轨基准 (CSEDB) 用于评估临床环境中的LLM.
- 使用CSEDB评估各种LLM的安全性和有效性表现.
- 发现绩效差异,特别是在高风险的临床场景中.
主要方法:
- 根据临床专家的共识制定了CSEDB框架,包括关键临床领域的30个指标.
- 由来自26个临床部门的32名专家医生生成了2069个开放式问答项目.
- 使用CSEDB测试了6个LLM,分析了一般和高风险场景中的性能.
主要成果:
- 课程的总体表现中等 (平均57.2%),安全性 (54.7%) 比有效性 (62.3%) 的得分更低.
- 在高风险场景中观察到显著的绩效下降 (13.3%) (p < 0.0001).
- 特定领域的医疗LLM优于通用模型,安全性 (0.912) 和有效性 (0.861) 分数更高.
结论:
- CSEDB提供了一种标准化指标,用于评估医学LLM的临床效用,风险暴露和需要改进的领域.
- 研究结果强调,在临床广泛采用之前,需要严格的安全性和有效性验证.
- 与一般模型相比,特定领域的LLM显示出更安全,更有效的医疗保健应用程序的潜力.
相关概念视频
Improving Translational Accuracy
14.0K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
14.0K
Improving Translational Accuracy
3.5K
3.5K
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
376
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
376
Sensitivity, Specificity, and Predicted Value
1.2K
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
1.2K


