Theoretical false positive psychology.

Area of Science:

• Scientific methodology
• Research integrity
• Psychological science

Background:

• Scientific research aims to produce authentic discoveries (true positives).
• Statistical significance is often interpreted as a true positive, but the null hypothesis (δ=0) may rarely be strictly true.
• Nuisance factors can introduce small, theoretically uninteresting effects.

Purpose of the Study:

• To examine the impact of increasing statistical power on true and false positives in scientific research.
• To evaluate the implications of increasing sample size (N) for theory-focused versus measurement-focused research.
• To determine how to maximize the positive predictive value at the theoretical level.

Main Methods:

• Conceptual analysis of statistical significance, effect size (δ), and statistical power.
• Distinction between statistical true positives and theoretical false positives.
• Evaluation of the role of sample size (N) in measurement-focused versus theory-focused research.

Main Results:

• Increasing statistical power (via larger N) enhances true positives at the effect size level.
• However, excessive power increases false positives at the theoretical level, as nuisance effects become significant.
• Theory-focused research risks accumulating findings based on theoretically unimportant effects with high power.

Conclusions:

• Simply increasing sample size (N) indefinitely is not universally beneficial for scientific progress.
• An optimal sample size (N) is necessary to balance statistical significance with theoretical relevance.
• Overemphasis on power can lead to a foundation of nuisance effects masquerading as discoveries, undermining scientific validity.