Jove
Visualize
Contact Us

Related Concept Videos

Uncertainty: Confidence Intervals00:54

Uncertainty: Confidence Intervals

3.1K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
3.1K
Uncertainty: Overview00:59

Uncertainty: Overview

528
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
528
Estimating Population Mean with Unknown Standard Deviation01:22

Estimating Population Mean with Unknown Standard Deviation

7.6K
In practice, we rarely know the population standard deviation. In the past, when the sample size was large, this did not present a problem to statisticians. They used the sample standard deviation s as an estimate for σ and proceeded as before to calculate a confidence interval with close enough results. However, statisticians ran into problems when the sample size was small. A small sample size caused inaccuracies in the confidence interval.
William S. Gosset (1876–1937) of the...
7.6K
Propagation of Uncertainty from Systematic Error01:10

Propagation of Uncertainty from Systematic Error

489
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
489
Propagation of Uncertainty from Random Error00:59

Propagation of Uncertainty from Random Error

655
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
655
Distributions to Estimate Population Parameter01:26

Distributions to Estimate Population Parameter

4.0K
The accurate values of population parameters such as population proportion, population mean, and population standard deviation (or variance) are usually unknown. These are fixed values that can only be estimated from the data collected from the samples. The estimates of each of these parameters are sample proportion, the sample mean, and sample standard deviation (or variance). To obtain the values of these sample statistics, data are required that have particular distribution and central...
4.0K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

The LEADING guideline: Reporting standards for expert panel, best-estimate diagnosis, and longitudinal expert all data (LEAD) methods.

Comprehensive psychiatry·2025
Same author

The LEADING Guideline: Reporting Standards for Expert Panel, Best-Estimate Diagnosis, and Longitudinal Expert All Data (LEAD) Studies.

medRxiv : the preprint server for health sciences·2024
Same author

Quantifying gender bias towards politicians in cross-lingual language models.

PloS one·2023
Same author

Quantifying gender biases towards politicians on Reddit.

PloS one·2022
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jun 12, 2025

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment
08:25

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

Published on: May 7, 2019

8.9K

Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift.

Dustin Wright1, Isabelle Augenstein1

  • 1University of Copenhagen, Department of Computer Science, Copenhagen, Denmark.

Plos One
|June 9, 2025
PubMed
Summary

Aggregating crowd-sourced labels using a simple average improves machine learning model performance and uncertainty estimation across diverse tasks, especially for subjective data. This method offers consistent results compared to individual soft-labeling techniques.

More Related Videos

Proteome-wide Quantification of Labeling Homogeneity at the Single Molecule Level
08:29

Proteome-wide Quantification of Labeling Homogeneity at the Single Molecule Level

Published on: April 19, 2019

6.1K
Author Spotlight: Deciphering Electrical Networks Behind Complex Brain Activities and Disorders
05:49

Author Spotlight: Deciphering Electrical Networks Behind Complex Brain Activities and Disorders

Published on: November 1, 2024

728

Related Experiment Videos

Last Updated: Jun 12, 2025

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment
08:25

Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment

Published on: May 7, 2019

8.9K
Proteome-wide Quantification of Labeling Homogeneity at the Single Molecule Level
08:29

Proteome-wide Quantification of Labeling Homogeneity at the Single Molecule Level

Published on: April 19, 2019

6.1K
Author Spotlight: Deciphering Electrical Networks Behind Complex Brain Activities and Disorders
05:49

Author Spotlight: Deciphering Electrical Networks Behind Complex Brain Activities and Disorders

Published on: November 1, 2024

728

Area of Science:

  • Machine Learning
  • Artificial Intelligence
  • Data Science

Background:

  • Expert annotations for machine learning are costly, while crowd-sourced labels can be unreliable.
  • Learning from crowd label distributions (soft labels) shows promise for performance and uncertainty estimation.
  • Existing studies primarily focus on in-domain settings with limited soft-labeling methods.

Purpose of the Study:

  • To conduct a large-scale empirical study of soft-labeling methods for crowd-sourced data in out-of-domain settings.
  • To evaluate 8 different soft-labeling methods across 4 language and vision tasks.
  • To propose and validate a simple averaging aggregation method for soft labels.

Main Methods:

  • Systematic analysis of 8 soft-labeling methods on 4 diverse language and vision tasks.
  • Implementation of a simple averaging strategy to aggregate soft labels.
  • Comparison of the aggregation method against individual soft-labeling methods and majority voting.

Main Results:

  • Averaging soft labels consistently improves predictive uncertainty estimation across most settings.
  • The proposed aggregation method maintains competitive raw performance compared to other approaches.
  • Method selection is less critical with abundant or minimal data, but aggregation significantly boosts uncertainty for subjective labels with moderate data.

Conclusions:

  • Simple averaging of soft labels provides a robust and consistent approach for learning from crowd-sourced annotations.
  • This aggregation strategy enhances model uncertainty estimation, particularly valuable for subjective tasks.
  • The findings offer practical guidance for selecting and applying crowd-labeling techniques in machine learning.