Trial-level noise and the reliability of the perceptual load effect: a preliminary investigation

The perceptual load effect, where increased perceptual demands reduce distractor processing, is a well-established phenomenon in attention research. Despite its robustness at the group level, its psychometric properties remain largely unexplored, limiting its utility in individual differences research. This gap is particularly important given growing concerns about the poor reliability of many canonical cognitive effects, which undermines their suitability for correlational studies. In this study, we assess whether hierarchical Bayesian models can provide precise estimates of the test-retest reliability of perceptual load sensitivity. A two-session design was employed to obtain empirical data, which was used to estimate trial-level noise. This noise estimate anchored a set of simulations examining how factors such as noise magnitude and trial count influence the precision of reliability estimates derived from hierarchical Bayesian modeling. While hierarchical Bayesian models offer a promising framework for disentangling trial-level noise from true inter-subject variation, our findings suggest that we still lack a clear picture of the perceptual load effect's true reliability when constrained to trial numbers that are practical for correlational research, at least when only two levels of load are present in the task. These results indicate that the reliability challenges observed in other cognitive control paradigms extend to the perceptual load effect and highlight the need for methodological innovations to improve its measurement in individual differences research.