How do my distributions differ? significance testing for the overlapping index using permutation tests.

Area of Science:

• Psychological Science
• Statistical Methodology
• Quantitative Psychology

Background:

• Traditional psychological research often uses statistical tests focused on single parameters like means.
• Empirical data frequently exhibit complex differences in variance, skewness, and shape, which are overlooked by mean-centric tests.
• There is a growing need for robust, assumption-light statistical methods in psychological research.

Purpose of the Study:

• Introduce the zeta-ov (ζov) test, a novel permutation-based inferential procedure.
• Evaluate the performance of the ζov test against traditional statistical tests using simulations.
• Promote distribution-aware statistical inference in psychological research.

Main Methods:

• Developed the ζov test based on the Overlapping Index, an effect size measuring distributional similarity.
• Conducted simulations varying mean, variance, skewness, and sample size to assess Type I error control and sensitivity.
• Compared the ζov test with t-tests, Welch's t-test, Wilcoxon-Mann-Whitney, Kolmogorov-Smirnov, and variance tests.

Main Results:

• The ζov test demonstrated adequate Type I error control across simulated scenarios.
• The ζov test showed high sensitivity to distributional differences, especially those involving multiple parameters.
• An applied example highlighted the ability of the ζov test to detect differences missed by mean-based analyses.

Conclusions:

• The ζov test provides a theoretically aligned global assessment of distributional differences without parametric assumptions.
• This method complements traditional tests by encouraging distribution-aware inference and integrating visualization.
• The ζov framework supports the methodological shift towards robust and interpretable statistical reasoning in the psychological sciences.