The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time...

Area of Science:

• Neuroscience
• Cognitive Psychology
• Psychometrics

Background:

• Reliability estimation is crucial for robust inference in scientific research.
• It is underutilized in neuroscience and cognitive psychology, hindering statistical power and reproducibility.
• Accurate reliability calculation for experimental learning tasks presents challenges.

Purpose of the Study:

• To highlight issues in reliability estimation for experimental learning tasks.
• To estimate internal consistency and split-half reliability for a widely used learning task.
• To demonstrate the impact of pre-processing, task length, and sample size on reliability.

Main Methods:

• Estimation of multiple internal consistency and split-half reliability metrics.
• Analysis of a large sample of 180 subjects performing the Alternating Serial Reaction Time Task.
• Investigation of factors influencing reliability, including pre-processing choices, task duration, and sample size.

Main Results:

• The Alternating Serial Reaction Time Task demonstrates respectable reliability, particularly when learning scores use reaction times and two-stage averaging.
• A task length of 25 blocks can achieve minimally acceptable reliability thresholds.
• Pre-processing choices, task length, and sample size significantly affect reliability estimation.

Conclusions:

• Reliability estimation enhances statistical power, effect sizes, and reproducibility while reducing measurement error.
• Multiple reliability metrics and their uncertainties provide a more comprehensive psychometric characterization than single estimates.
• The Alternating Serial Reaction Time Task is a reliable measure, especially under optimized scoring and length conditions.