校准作为衡量生物医学自然语言处理中大型语言模型可靠性的研究
Rodrigo de Oliveira1, Matthew Garber2, James M Gwinnutt3
1IQVIA - Real World Solutions, London, W2 1AF, United Kingdom.
JAMIA open
|July 14, 2025
概括
大型语言模型 (LLM) 显示生物医学任务的校准不佳,影响了可信度. 临时校准技术显著提高了可靠性,使LLM在现实应用中更加可靠.
科学领域:
- 生物医学自然语言处理 (BioNLP)
- 机器学习校准的机器学习
背景情况:
- 评估大型语言模型 (LLM) 的可靠性对于它们在生物医学自然语言处理 (BioNLP) 等敏感领域的可靠部署至关重要.
- 现有的校准指标可能无法完全捕捉基于语言的任务的细微差别,需要更强大的评估方法.
研究的目的:
- 通过使用生物医学语言理解和推理基准 (BLURB) 来评估9个LLM在各种BioNLP任务中的校准.
- 引入并使用灵活预期校准误差 (Flex-ECE) 来更现实地评估LLM性能,并考虑部分正确性.
- 调查不同信心评分策略和后期校准技术的有效性.
主要方法:
- 收集了来自9个LLM在6个任务中的13个BLURB数据集的响应和信心评分.
- 采用口头,自我一致性和混合策略进行信心评分.
- 引入Flex-ECE作为一种新的评估指标,并将其与传统方法进行比较.
- 应用同位素回归和直方图组分作为后期校准技术.
主要成果:
- LLM表现出较差的即时校准,平均校准在任务中从23.9%到46.6%不等.
- 医学Llama3-8B表现出最好的整体校准 (29.8%),而Flan-T5-XXL在多个数据集中排名最高.
- 与口头 (42.0%) 和混合 (44.2%) 相比,自我一致策略的校准率更好 (27.3%).
- 后期校准方法大大改善了校准,将Flex-ECE的平均值降至0.1%-4.1%.
结论:
- 法律法规的固有不良校准对其在现实世界BioNLP场景中的应用构成重大风险.
- 后期校准是提高法学士的可信度的一个重要和推的做法.
- Flex-ECE指标提供了对LLM性能更现实的评估,特别是对于部分正确性可能存在的模型.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
587
06:18Measurement of Specific Mycobacterial Mistranslation Rates with Gain-of-function Reporter Systems
Published on: April 26, 2019
6.1K
相关概念视频
Instrument Calibration
273
Instrument calibration is essential for ensuring that instruments produce accurate and consistent results. It is vital in manufacturing, healthcare, testing laboratories, and scientific research. Calibration processes are specific to each instrument and help enhance data accuracy. Each instrument has a unique calibration process tailored to its design and function to improve data accuracy.
Analytical Balance Calibration
An analytical balance measures mass and requires regular calibration to...
Analytical Balance Calibration
An analytical balance measures mass and requires regular calibration to...
273
Improving Translational Accuracy
11.9K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.9K
Calibration Curves: Correlation Coefficient
2.4K
In a linear calibration curve, there is a value called the calibration coefficient, denoted by 'r,' which measures the strength and the direction of association between two variables. The correlation coefficient value ranges from −1 to +1. A value of +1 indicates a perfect positive linear correlation, −1 denotes a perfect negative correlation, and 0 implies no correlation between the two variables. A positive correlation value establishes that as one variable increases, the...
2.4K
Calibration Curves: Linear Least Squares
2.3K
A calibration curve is a plot of the instrument's response against a series of known concentrations of a substance. This curve is used to set the instrument response levels, using the substance and its concentrations as standards. Alternatively, or additionally, an equation is fitted to the calibration curve plot and subsequently used to calculate the unknown concentrations of other samples reliably.
For data that follow a straight line, the standard method for fitting is the linear...
For data that follow a straight line, the standard method for fitting is the linear...
2.3K
Accuracy and Precision
11.2K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value. Highly accurate...
11.2K
Uncertainty in Measurement: Accuracy and Precision
81.6K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.
81.6K
