The Pervasive Problem of Post Hoc Data Selection in Studies on Unconscious Processing

Area of Science:

• Cognitive Psychology
• Neuroscience
• Research Methodology

Background:

• Studies on unconscious mental processes often involve stimuli participants are unaware of.
• Methodological challenges arise when participants become aware of critical stimuli during experiments.
• Post hoc selection of participants based on awareness measures is debated regarding statistical validity.

Purpose of the Study:

• To explain the statistical problem of regression-to-the-mean artifacts in unconscious processing research.
• To refute claims that this statistical artifact's influence is overestimated.
• To argue against the use of post hoc data selection without safeguards.

Main Methods:

• Explanation of the regression-to-the-mean artifact with a concrete example.
• Analysis and refutation of arguments presented by Sklar et al. (2021).
• Discussion of safeguards necessary for data selection in unconscious processing studies.

Main Results:

• Demonstration of how post hoc participant selection can lead to regression-to-the-mean artifacts.
• Identification of fundamental mischaracterizations of the statistical problem by Sklar et al.
• Refutation of the claim that the artifact's influence is overestimated.

Conclusions:

• Post hoc data selection in studies of unconscious processing is prone to significant statistical artifacts.
• The method, as applied by Sklar et al., risks unjustified rejection of genuine unconscious effects.
• Without appropriate safeguards, post hoc data selection should not be used in research on unconscious processing.