Illusory inconsistencies in judgment: Stimulus-evoked reference sets and between-subjects designs

Asked to judge the subjective size of numbers in a between-subjects design, participants rated 9 as larger than 221 (Birnbaum, 1999). The 9 > 221 effect seems to indicate that different stimuli evoke different contexts for comparison, and sounds a warning for the interpretation of between-subjects comparisons. We show that, contrary to appearances, the effect is not a result of stimulus-evoked reference sets. Instead, it is an artifact of the original 1-10 response scale and task instructions, which encourage a conflation of the response scale and the reference set. When ratings are expressed on a 1-1000 scale, or on a non-numerical slider scale, the effect reverses. However, we also show that stimuli can evoke their own comparative contexts, generating illusions of inconsistency in between-subjects designs. We report two novel findings - a 9 > 009 effect and a -2 > 2 effect - which are best explained by stimulus-evoked reference sets. Thus, while revealing that the 9 > 221 effect is an artifact of the original response scale, our study ultimately affirms Birnbaum's warning about the comparison of between-subjects ratings.