Stimulus sampling reimagined: Designing experiments with mix-and-match, analyzing results with stimulus plots

Area of Science:

• Psychology
• Experimental Design
• Research Methodology

Background:

• Stimuli selection in psychological research is often unsystematic, leading to irreproducible experiments and undetected confounds.
• Current statistical analyses aggregate data across stimuli, masking potential stimulus-specific effects and confounds.

Purpose of the Study:

• To introduce systematic and reproducible methods for stimuli selection and analysis in psychology experiments.
• To enhance the detection of confounds and improve the overall rigor of psychological research.

Main Methods:

• The study proposes the "Mix-and-Match" procedure for systematic, reproducible, and stratified stimulus sampling.
• Introduction of "Stimulus Plots" for visualizing and reporting stimulus-level results, comparing observed versus expected variation.
• Application of these innovations to published studies to demonstrate their impact.

Main Results:

• The proposed methods reveal how reimagined stimulus sampling and analysis can alter findings from published psychological studies.
• Demonstration of improved detection of confounds and enhanced experimental transparency.

Conclusions:

• Adopting systematic stimulus sampling and stimulus-level analysis significantly improves the reproducibility and validity of psychological research.
• The proposed "Mix-and-Match" procedure and "Stimulus Plots," along with a disclosure form, offer a framework for more rigorous experimental practices.