Reaction time analysis with outlier exclusion: bias varies with sample size

Area of Science:

• Cognitive Psychology
• Psychometrics
• Behavioral Science

Background:

• Investigative researchers often use restricted means to mitigate the impact of outlier response times.
• This method involves excluding response times exceeding a set standard deviation threshold (e.g., 2.0, 2.5, or 3.0) from the mean.
• However, the skewed nature of reaction time distributions can lead to inherent biases in this approach.

Purpose of the Study:

• To investigate the potential bias introduced by calculating restricted means, particularly in the context of skewed reaction time distributions.
• To examine how sample size differences across conditions exacerbate this bias, leading to differential bias.
• To quantify the impact of differential bias on the comparison of experimental conditions.

Main Methods:

• Simulations were employed to model reaction time data with skewed distributions.
• Restricted means were computed using various standard deviation cutoffs (2.0, 2.5, 3.0).
• Differential bias was assessed by comparing conditions with differing numbers of observations (e.g., <10 vs. >20).

Main Results:

• The computation of restricted means systematically underestimates the true population average response time due to distribution skewness.
• Differential bias significantly impacts comparisons between conditions with varying sample sizes, especially when sample sizes are small (<10) versus larger (>20).
• Strongly skewed distributions and higher standard deviation cutoffs (e.g., 3.0) amplify differential bias, affecting comparisons even with larger sample sizes.

Conclusions:

• The use of restricted means, while intended to improve data quality, introduces a systematic underestimation bias in reaction time research.
• Differential bias poses a significant threat to the validity of findings when comparing conditions with unequal sample sizes, particularly in skewed data.
• Researchers should exercise caution when interpreting restricted means, especially in studies with varying sample sizes and skewed reaction time distributions, to avoid erroneous conclusions.