Does Increasing Sample Size Inevitably Lead to Statistical Significance? Insights From a Monte Carlo Simulation Study...

Area of Science:

• Statistics
• Biostatistics
• Research Methodology

Background:

• A common misconception posits that larger sample sizes automatically yield smaller P-values and increased statistical power.
• This study investigates the relationship between sample size and statistical significance under varying conditions.

Purpose of the Study:

• To examine the effect of sample size on P-values and statistical power using Monte Carlo simulations.
• To clarify whether larger sample sizes can generate significance in the absence of a true effect.

Main Methods:

• Monte Carlo simulations with 1000 iterations for 11 sample sizes (n=5-800).
• Two scenarios simulated: null hypothesis true (no effect) and alternative hypothesis true (effect present).
• Calculated absolute mean difference, standard error, P-value, and proportion of significant results (P < 0.05).

Main Results:

• When a true effect existed, statistical power increased significantly with sample size (5.4% to 98.1%).
• Without a true effect, P-values remained uniformly distributed, with significant results consistently near the 5% nominal level.
• Increasing sample size under the null hypothesis did not inflate the rate of significant findings.

Conclusions:

• Larger sample sizes do not manufacture statistical significance if no true effect is present.
• Increased sample size solely improves the detection of existing effects.
• Study design and interpretation should prioritize effect size and clinical relevance over sample size alone.