Challenging the reliability and validity of cognitive measures: the case of the numerical distance effect

The numerical distance effect (NDE) is one of the most robust effects in the study of numerical cognition. However, the validity and reliability of distance effects across different formats and paradigms has not been assessed. Establishing whether the distance effect is both reliable and valid has important implications for the use of this paradigm to index the processing and representation of numerical magnitude in both behavioral and neuroimaging studies. In light of this, we examine the reliability and validity of frequently employed variants (and one new variant) of the numerical comparison task: two symbolic comparison variants and two nonsymbolic comparison variants. The results of two experiments demonstrate that measures of the NDE that use nonsymbolic stimuli are far more reliable than measures of the NDE that use symbolic stimuli. With respect to correlations between measures, we find evidence that the NDE that arises using symbolic stimuli is uncorrelated with the NDE that is elicited by using nonsymbolic stimuli. Results are discussed with respect to their implications for the use of the NDE as a metric of numerical processing and representation in research with both children and adults.