Does Increasing Sample Size Inevitably Lead to Statistical Significance? Insights From a Monte Carlo Simulation Study...

This study addresses the common misconception that larger sample sizes inevitably produce smaller P-values and higher statistical power, using Monte Carlo simulations to examine the effect of sample size under scenarios with and without a true effect.

Monte Carlo simulations were performed with 1000 iterations for each of 11 sample sizes per group (n = 5-800). Two scenarios were simulated: (1) both groups drawn from the same normal distribution (μ = 4.5, σ = 0.5; null hypothesis H₀ true), and (2) groups drawn from different normal distributions (μ₁= 4.5, σ = 0.5; μ₂= 4.4, σ = 0.5; alternative hypothesis H₁ true). For each iteration, the absolute mean difference (|X̅₁ - X̅₂|), standard error (S_e), t-test p-value and the proportion of significant results (p < 0.05) were calculated. This proportion was interpreted as the Type I error rate or statistical power, depending on the scenario.

When a true effect existed, increasing sample size led to a marked rise in statistical power, with the proportion of significant results increasing from 5.4% to 98.1%. In contrast, when no true effect existed, p-values remained uniformly distributed across all sample sizes, and the frequency of p < 0.05 consistently approximated the nominal 5% significance level. Importantly, increasing sample size under the null hypothesis did not increase the likelihood of detecting significance.

Larger sample sizes do not create statistical significance in the absence of a true effect; they only enhance the detection of effects that genuinely exist. Study design and statistical interpretation should therefore be guided by effect size and clinical relevance rather than by sample size alone.