Illusory inconsistencies in judgment: Stimulus-evoked reference sets and between-subjects designs

Area of Science:

• Cognitive Psychology
• Judgment and Decision Making
• Experimental Psychology

Background:

• Between-subjects designs in psychological research can lead to unexpected findings.
• The '9 > 221' effect, where 9 was rated larger than 221, initially suggested stimulus-evoked comparison contexts.
• This effect raised concerns about the validity of interpreting between-subjects ratings.

Purpose of the Study:

• To investigate the cause of the '9 > 221' effect in subjective number size judgments.
• To determine if the effect stems from stimulus-evoked reference sets or response scale artifacts.
• To explore novel findings like '9 > 009' and '-2 > 2' effects and their explanations.

Main Methods:

• Participants judged the subjective size of numbers in a between-subjects design.
• The study manipulated response scales (1-10, 1-1000, slider scale) to assess their impact.
• Novel stimuli and conditions were introduced to test stimulus-evoked reference set hypotheses.

Main Results:

• The '9 > 221' effect was found to be an artifact of the original 1-10 response scale and instructions.
• Reversing the effect occurred when using a 1-1000 scale or a non-numerical slider.
• Two new effects, '9 > 009' and '-2 > 2', were observed and explained by stimulus-evoked reference sets.

Conclusions:

• The '9 > 221' effect is an artifact, not evidence of stimulus-evoked contexts for that specific instance.
• Stimuli can indeed evoke comparative contexts, leading to apparent inconsistencies in between-subjects designs.
• The study validates the original warning about the challenges of interpreting between-subjects ratings.