Related Experiment Video
Updated: Jun 29, 2025
10:22
Split Point Analysis and Uncertainty Quantification of Thermal-Optical Organic/Elemental Carbon Measurements
Published on: September 7, 2019
8.3K
Uncertainty limits the use of power analysis
Jolynn Pek1, Mark A Pitt1, Duane T Wegener1
1Department of Psychology, The Ohio State University.
Journal of Experimental Psychology. General
|April 8, 2024
Summary
Statistical power calculations for experiment design can be misleading. Uncertainty in effect size estimates and random fluctuations in population effect size make power values unreliable for justifying sample size.
Area of Science:
- Statistics
- Experimental Design
- Psychology
Background:
- Statistical power is commonly used to justify sample size in experimental design.
- Classical power calculations often neglect crucial sources of uncertainty.
- This can lead to a false sense of precision in design choices.
Purpose of the Study:
- To investigate the impact of uncertainty on statistical power calculations.
- To demonstrate the consequences of incorporating sampling variability and population effect size fluctuations.
- To evaluate the reliability of power-based sample size justifications.
Main Methods:
- Utilized simulation studies to model the effects of uncertainty.
- Incorporated sampling variability in the estimation of effect size (Cohen's d).
- Introduced random fluctuations in the population effect size.
Main Results:
- Sampling variability in effect size estimates introduces substantial uncertainty in power and sample size determination.
- Random fluctuations in population effect size can render calculated power values meaningless.
- Simulations showed that calculated power values can be highly unstable and misleading.
Conclusions:
- Researchers should place minimal confidence in power-based sample size justifications.
- Classical power calculations that ignore uncertainty provide a false sense of precision.
- Rethinking the reliance on traditional power analysis for experimental design is recommended.
More Related Videos
Related Concept Videos
00:59Propagation of Uncertainty from Random Error
681
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
681
00:59Uncertainty: Overview
552
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
552
04:08The Uncertainty Principle
23.3K
Werner Heisenberg considered the limits of how accurately one can measure properties of an electron or other microscopic particles. He determined that there is a fundamental limit to how accurately one can measure both a particle’s position and its momentum simultaneously. The more accurate the measurement of the momentum of a particle is known, the less accurate the position at that time is known and vice versa. This is what is now called the Heisenberg uncertainty principle. He...
23.3K
01:10Propagation of Uncertainty from Systematic Error
517
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
517
00:54Uncertainty: Confidence Intervals
4.1K
The confidence interval is the range of values around the mean that contains the true mean. It is expressed as a probability percentage. The interpretation of a 95% confidence interval, for instance, is that the statistician is 95% confident that the true mean falls within the interval. The upper and lower limits of this range are known as confidence limits. The confidence limits for the true mean are estimated from the sample's mean, the standard deviation, and the statistical factor...
4.1K
03:37Uncertainty in Measurement: Accuracy and Precision
73.7K
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.
73.7K