Multiple trials may yield exaggerated effect size estimates
Andrew Brand1, Michael T Bradley, Lisa A Best
1Department of Psychology, Kings College, London.
Abstract:
Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately, and thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this article describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items, but remain substantial in some cases. Some concepts, such as a P300 wave or a test score, are best defined as a composite of measures. Troubles may arise in more exploratory research where the interrelations among trials or items may not be well described.
Related Concept Videos
One-Way ANOVA: Equal Sample Sizes
Different sample means can result in different values for the variance estimate: variance between samples. This is because the variance between samples is calculated as the product of the sample size and the variance between the...
Testing a Claim about Population Proportion
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
Regression Toward the Mean
One-Way ANOVA: Unequal Sample Sizes
Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...


