Properties of hypothesis testing techniques and (Bayesian) model selection for exploration-based and theory-based...

Area of Science:

• Statistics
• Psychology
• Social Sciences

Background:

• Comparing means is crucial in various scientific fields.
• Existing methods include hypothesis testing and model selection.
• Hypotheses can be exploration-based (equality constraints) or theory-based (order restrictions).

Purpose of the Study:

• To evaluate the performance of six techniques for comparing means.
• To compare hypothesis testing, information criteria, and Bayesian model selection.
• To assess the impact of exploration vs. confirmation and equality vs. order restrictions.

Main Methods:

• A simulation study was conducted.
• Six techniques were examined, varying method (testing, model selection) and hypothesis set (exploration, confirmation).
• Performance was evaluated under conditions of homogeneity and heterogeneity of variances, with equal and unequal sample sizes.

Main Results:

• Confirmatory (theory-based) hypothesis investigation is advantageous over exploration.
• Order-restricted hypotheses provide more power than equality restrictions.
• Model selection outperforms hypothesis testing for multiple theory-based hypotheses.
• Techniques are robust to variance heterogeneity with equal sample sizes.
• Performance degrades with unequal sample sizes and variance heterogeneity, influenced by effect size and variance trends.

Conclusions:

• Theory-based, order-restricted hypothesis testing with model selection is recommended for comparing means.
• Researchers should consider the interplay of sample size and variance heterogeneity.
• The alignment of group variances and sample sizes can mitigate performance issues.