从人群注释中汇总软标签可以改善分布转移下的不确定性估计
Dustin Wright1, Isabelle Augenstein1
1University of Copenhagen, Department of Computer Science, Copenhagen, Denmark.
PloS one
|June 9, 2025
概括
使用简单的平均值汇总众包标签可以提高机器学习模型的性能和在各种任务中估计不确定性,特别是在主观数据中. 与单个软标签技术相比,这种方法提供了一致的结果.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 数据科学数据科学数据科学
背景情况:
- 专家对机器学习的注释是昂贵的,而众包标签可能不可靠.
- 从群众标签分发 (软标签) 中的学习显示出对性能和不确定性估计的希望.
- 现有的研究主要集中在具有有限软标签方法的域内设置上.
研究的目的:
- 在域外环境中对众筹数据进行软标签方法的大规模实证研究.
- 在4个语言和视觉任务中评估8种不同的软标签方法.
- 提出和验证软标签的简单平均聚合方法.
主要方法:
- 对8种软标签方法对4种不同的语言和视觉任务进行系统分析.
- 实施一个简单的平均化策略来汇总软标签.
- 聚合方法与个人软标签方法和多数投票的比较.
主要成果:
- 软标签的平均值在大多数设置中始终改善预测不确定性估计.
- 与其他方法相比,拟议的聚合方法保持了具有竞争力的原始性能.
- 方法选择在数据丰富或最小的情况下不那么重要,但聚合在数据中等的情况下显著增加了主观标签的不确定性.
结论:
- 软标签的简单平均值提供了一个强大而一致的方法,用于从众包注释中学习.
- 这种聚合策略增强了模型不确定性估计,对于主观任务特别有价值.
- 这些发现为在机器学习中选择和应用众包标签技术提供了实际指导.
相关概念视频
Uncertainty: Confidence Intervals
3.1K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
3.1K
Uncertainty: Overview
528
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
528
Estimating Population Mean with Unknown Standard Deviation
7.6K
In practice, we rarely know the population standard deviation. In the past, when the sample size was large, this did not present a problem to statisticians. They used the sample standard deviation s as an estimate for σ and proceeded as before to calculate a confidence interval with close enough results. However, statisticians ran into problems when the sample size was small. A small sample size caused inaccuracies in the confidence interval.
William S. Gosset (1876–1937) of the...
William S. Gosset (1876–1937) of the...
7.6K
Propagation of Uncertainty from Systematic Error
489
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
489
Propagation of Uncertainty from Random Error
655
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
655
Distributions to Estimate Population Parameter
4.0K
The accurate values of population parameters such as population proportion, population mean, and population standard deviation (or variance) are usually unknown. These are fixed values that can only be estimated from the data collected from the samples. The estimates of each of these parameters are sample proportion, the sample mean, and sample standard deviation (or variance). To obtain the values of these sample statistics, data are required that have particular distribution and central...
4.0K


