The Danger of Testing by Selecting Controlled Subsets, with Applications to Spoken-Word Recognition

Area of Science:

• Psycholinguistics
• Statistical Methodology
• Experimental Design

Background:

• Researchers often dichotomize continuous variables to analyze effects on outcomes.
• This approach is sometimes used to control for confounding covariates via subset selection.
• The
• divide, select, and test
• method is prevalent in psycholinguistics literature.

Purpose of the Study:

• To investigate the potential for erroneous results arising from dichotomizing continuous variables.
• To demonstrate how subset selection in dichotomization can lead to false positive or false negative findings.
• To re-evaluate previously published conclusions using this methodology.

Main Methods:

• Analysis of the
• divide, select, and test
• methodology in statistical analysis.
• Examining the impact of specific subset selection strategies on dichotomized data.
• Replication and re-analysis of published findings on lexical properties and spoken-word recognition.

Main Results:

• Dichotomization, particularly with certain subset selection methods, can create spurious relationships between variables.
• Previously reported effects of lexical properties on spoken-word recognition may not hold under this methodology.
• Randomly selecting balanced subsets frequently masks existing relationships between variables.

Conclusions:

• The dichotomization approach, despite its appeal for controlling confounders, can yield unreliable results.
• Researchers must be aware of the implicit freedoms in experimental design that can lead to false conclusions.
• This study highlights a critical issue in statistical practices, urging caution in interpreting findings derived from dichotomized data.