A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests

Area of Science:

• Cognitive psychology
• Behavioral science
• Statistical methodology

Background:

• Experimental research in behavior and cognition faces challenges in controlling all subject and stimulus characteristics.
• Nuisance variables (e.g., intelligence, word frequency) are often correlated with primary variables of interest, complicating interpretation.
• A common but flawed practice involves using inferential tests to 'match' groups on these nuisance variables.

Purpose of the Study:

• To critically evaluate the appropriateness of using inferential tests for controlling nuisance variables in experimental research.
• To highlight the conceptual and practical limitations of this widespread statistical practice.
• To propose and briefly discuss regression analysis as a more suitable alternative.

Main Methods:

• Conceptual analysis of statistical test interpretation and application.
• Survey of current research practices regarding the control of nuisance variables.
• Discussion of regression analysis as an alternative methodology.

Main Results:

• Inferential testing for nuisance variable control is statistically inappropriate and philosophically misguided.
• This flawed practice is prevalent in experimental research, indicating a common misunderstanding of statistical inference.
• The misuse of inferential tests can lead to erroneous conclusions about group comparability.

Conclusions:

• The practice of using inferential tests to establish matching on nuisance variables should be abandoned.
• Researchers should adopt more appropriate statistical methods, such as regression analysis, to account for confounding variables.
• Correctly applying statistical methods is crucial for the validity and interpretability of experimental findings in behavioral and cognitive sciences.