一个用于评估临床研究声明的大型语言模型的数据集
Boya Zhang1, Alban Bornet2, Anthony Yazdani2
1Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, Switzerland. boya.zhang@unige.ch.
大型语言模型 (LLM) 对健康主张验证有希望,但需要仔细审查. 来自临床试验的CliniFact数据集,基准了LLM的表现,歧视型模型的表现优于生成型模型.
科学领域:
- 医疗信息学 医疗信息学
- 医疗保健中的人工智能
- 临床试验分析
背景情况:
- 大型语言模型 (LLM) 提供了验证健康主张的潜力.
- 挑战包括在医疗保健中的LLM幻觉和逻辑陈述理解.
- 对LLM的审查对于可靠的临床应用至关重要.
研究的目的:
- 介绍CliniFact,一个用于评估临床研究索赔验证LLM绩效的新型数据集.
- 基准LLM使用临床研究中的假设测试结果.
- 评估在这一领域的歧视性和生成性LLM的能力.
主要方法:
- 从临床试验数据构建CliniFact,包括干预措施,结果和结果.
- 与科学出版物中的支持证据链接衍生主张.
- 评估了LLM,包括BioBERT和Llama3-70B,与CliniFact数据集对比.
主要成果:
- CliniFact包括来自992个试验和1540个出版物的1970个实例.
- 与生成型模型相比 (例如,Llama3-70B在53.6%),歧视型模型的准确性更高 (例如,BioBERT在80.2%).
- 绩效的差异在统计学上是显著的 (p值<0.001).
结论:
- 在临床索赔验证中,CliniFact 作为一个有价值的基准来评估LLM准确性.
- 对此任务的歧视性模型表现出优于生成型模型的性能.
- 需要进一步发展,以提高医疗保健索赔验证中的LLM可靠性.
更多相关视频
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
06:48Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
Published on: June 25, 2019
相关概念视频
Clinical Trials
There are four phases in a clinical trial. A phase one...
Clinical Trials: Overview
Improving Translational Accuracy
Statistical Software for Data Analysis and Clinical Trials
Testing a Claim about Mean: Unknown Population SD
Estimating a population mean requires the samples to be approximately normally distributed. The data should be collected from the randomly selected samples having no sampling bias. There is no specific requirement for sample size. But if the sample size is less than 30, and we don't know the population standard deviation, a different approach is used;...
Testing a Claim about Mean: Known Population SD
Estimating a population mean requires the samples to be distributed normally. The data should be collected from the randomly selected samples having no sampling bias. The sample size needed to be higher than 30, and most importantly, the population standard deviation should be already known.
In most realistic situations, the population standard deviation is often unknown, but in rare circumstances, when it...
