CeRTS:在大型语言模型中检索确定性检索令牌,提取临床信息
Lars E Schimmelpfennig1, Kriti Bhattarai2, Inez Y Oh1
1Institute for Informatics, Data Science & Biostatistics, Washington University School of Medicine, St. Louis, MO, United States.
Journal of biomedical informatics
|August 25, 2025
概括
确定性检索令牌搜索 (CeRTS) 改善了临床数据提取的大型语言模型 (LLM) 的不确定性估计. 这种新方法为提取的信息提供了更可靠的信心分数,提高了LLM在医疗保健中的可行性.
科学领域:
- 人工智能
- 自然语言处理
- 临床信息学
背景情况:
- 大型语言模型 (LLM) 需要对临床应用进行可靠的不确定性估计,特别是从电子健康记录中提取信息.
- 现有的代币级不确定性估计器是有限的,因为它们只考虑单个输出序列中的概率.
研究的目的:
- 开发和评估使用LLM进行结构化信息提取的新型不确定性估计器Certainty Retrieval Token Search (CeRTS).
- 利用JSON输出结构约束来考虑所有可能的序列及其概率以更准确地衡量模型的可靠性.
主要方法:
- 在八个开源LLM上对金标准不确定性估计器进行了评估.
- 临床特征是从肺癌病例摘要中提取的.
- 使用校准 (布赖尔分数) 和区分 (AUROC) 来量化性能.
主要成果:
- 与之前的估计器相比,CeRTS在所有评估的LLM中都显示出更高的区分能力.
- 在大多数情况下,CeRTS实现了更好的校准,显示出更好的可靠性.
- 在Qwen-2.5模型中,模型的可靠性和准确性之间的一致性是最强的.
结论:
- CeRTS为从临床文本中提取基于LLM的信息提供了精确校准的信心评分,为研究人员提供了定量可靠性衡量标准.
- 虽然CeRTS通常是强大的,但它面临着DeepSeek-R1模型的挑战,可能是由于其链式思维.
- CeRTS方法适用于任何需要超出临床数据的可靠不确定性估计的领域.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
575
07:50A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
16.0K
相关概念视频
Clinical Trials
8.1K
Clinical trials are prospective experimental studies conducted on humans to determine the safety and efficacy of treatments, drugs, diet methods, and medical devices. Using statistics in clinical trials enables researchers to derive reasonable and accurate conclusions from the collected data, allowing them to make wise decisions in uncertain situations. In medical research, statistical methods are crucial for preventing errors and bias.
There are four phases in a clinical trial. A phase one...
There are four phases in a clinical trial. A phase one...
8.1K
Improving Translational Accuracy
2.7K
2.7K
Clinical Trials: Overview
3.4K
Clinical development focuses on how the drug will interact with the human body and encompasses four key phases of clinical trials, each serving a specific purpose in assessing the safety and effectiveness of new drugs. These phases overlap and build upon one another. Phase I involves a small group of healthy volunteers (typically 20-80 individuals) or, in cases where significant toxicity is expected, patients with the targeted disease, such as cancer or AIDS. The volunteers are tested for...
3.4K
Uncertainty: Confidence Intervals
4.7K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
4.7K
ER Retrieval Pathway
3.9K
In the secretory pathway, vesicles transport proteins from one cellular compartment to another in forward transport to deliver the protein to its correct location. Occasionally, misfolded proteins and incorrect proteins escape their original compartments, and a retrieval pathway is used to return the escaped proteins to their original compartment.
The ER uses many checkpoints to prevent the entry of incorrectly folded or a resident protein as cargo onto a transport vesicle. These mechanisms...
The ER uses many checkpoints to prevent the entry of incorrectly folded or a resident protein as cargo onto a transport vesicle. These mechanisms...
3.9K
Retrieval
169
Retrieval is the process of getting information out of memory storage and back into conscious awareness. This ability is essential for daily tasks like brushing hair and teeth, driving to work, and performing job duties. Retrieval occurs in three ways: recall, recognition, and relearning.
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...
169
