量化基于BERT的问答系统的信任转移,该系统对乱实例进行评估
1Information Sciences Institute, University of Southern California, Marina del Rey, California, United States of America.
PloS one
|December 20, 2023
概括
变压器模型在多选自然语言处理 (NLP) 任务中表现出色. 然而,他们对模两可的情况的信心,比如错误的答案选择,与预期的行为有很大不同,需要改进测试.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 基于变压器的神经网络在多选择自然语言处理 (NLP) 任务中取得了显著进展,包括问答 (QA).
- 在模糊的场景中对这些模型的系统评估,在模糊的场景中可能没有正确的答案,尽管在现实世界中具有相关性,但仍未得到充分探索.
研究的目的:
- 在模糊的情况下实验评估基于变压器的质量保证模型.
- 调查模型在没有正确答案选择的提示提示时如何表现.
主要方法:
- 设计了三个探测器,通过引入干扰来系统地"混"QA实例.
- 使用已建立的基于变压器的多选项质量保证系统进行实验.
- 在两个基准数据集上评估性能.
主要成果:
- 变压器模型表现出不同于模两可的质量保证场景中预期行为的信心水平.
- 模型对错误选择的信心表明在区分正确和错误的选择方面缺乏确定性.
- 在理想化质量保证任务上的高性能不能可靠地预测模两可的实例中的性能.
结论:
- 当前基于变压器的质量保证模型可能会在模两可的情况下扎,表现不同于预期.
- 模型无法可靠地区分模糊和明确的背景,突出了这些模型的局限性.
- 在将这些模型部署到具有人类影响的关键应用中之前,改进的测试协议和基准测试至关重要.
更多相关视频
相关概念视频
Uncertainty: Confidence Intervals
4.1K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
4.1K
Confidence Coefficient
7.6K
The confidence coefficient is also known as the confidence level or degree of confidence. It is the percent expression for the probability, 1-α, that the confidence interval contains the true population parameter assuming that the confidence interval is obtained after sufficient unbiased sampling; for example, if the CL = 90%, then in 90 out of 100 samples the interval estimate will enclose the true population parameter. Here α is the area under the curve, distributed equally under...
7.6K
Interpretation of Confidence Intervals
5.7K
A confidence interval is a better estimate of the population than a point estimate, as it uses a range of values from a sample instead of a single value.
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...
5.7K
Uncertainty in Measurement: Accuracy and Precision
73.7K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.
73.7K
Confidence Intervals
6.2K
An unbiased point estimate is often insufficient to predict a population estimate, such as population mean or population proportion. In this scenario, a confidence interval is used. A confidence interval is an estimate similar to a sample proportion. However, unlike the point estimate which is a single value, the confidence interval contains a range of values. These values have lower and upper limits, known as confidence limits, and can be designated as L1 and L2, respectively.
A...
A...
6.2K
Propagation of Uncertainty from Systematic Error
522
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
522


