Toward the validation of crowdsourced experiments for lightness perception.

Area of Science:

• Psychology
• Perception Science
• Human-Computer Interaction

Background:

• Crowdsourcing platforms are increasingly used for perceptual studies, including color perception.
• Concerns exist regarding the reliability of crowdsourced color studies due to uncontrolled participant environments.
• Stimulus presentation significantly impacts color perception, posing challenges for online research.

Purpose of the Study:

• To propose a method for mitigating errors in crowdsourced color perception studies.
• To validate a mathematical framework for estimating sample sizes in online perception research.
• To demonstrate that online crowdsourced studies can achieve comparable power to in-person studies.

Main Methods:

• Applied the Thurstonian model, treating crowdsourced experiments as a convolution of perceptual and stimulus presentation error distributions.
• Developed a mathematical formula to estimate the required sample size for equivalent statistical power.
• Replicated a large-scale crowdsourced lightness perception study using a controlled in-person methodology.

Main Results:

• The proposed mathematical model effectively accounts for variability in crowdsourced experimental designs.
• The replication study confirmed that online crowdsourced experiments can yield results comparable to controlled in-person studies.
• Sufficient sample sizes in online studies can compensate for the lack of environmental control.

Conclusions:

• Crowdsourced color vision studies are viable with adequate sample sizes, broadening access to diverse participant pools.
• The developed framework provides researchers with confidence in the validity of online versus in-person study designs.
• This research validates the use of online platforms for robust human perception research.