Beyond the traditional simulation design for evaluating type 1 error control: From the "theoretical" null to...

Area of Science:

• Genetics and Bioinformatics
• Statistical Methodology
• Computational Biology

Background:

• Evaluating new statistical tests requires rigorous type 1 error (T1E) control assessment via simulations.
• The "theoretical" null hypothesis (no association) is commonly used in simulations (design S0).
• Real-world genetic analyses operate under an "empirical" null, where most markers lack association with the outcome.

Purpose of the Study:

• To investigate the impact of different null simulation designs on statistical test evaluation.
• To compare the "theoretical" null (S0) with "empirical" null designs (S1.1, S1.2) in assessing T1E control.
• To highlight discrepancies in method accuracy revealed by distinct simulation approaches.

Main Methods:

• Simulation designs S0 (theoretical null) and S1.1/S1.2 (empirical null) were employed.
• The accuracy of a likelihood ratio test was examined using these simulation designs.
• Tests for variance heterogeneity, univariate, and multivariate interactions served as examples.

Main Results:

• Simulation design S0 indicated accurate T1E control for the likelihood ratio test.
• Designs S1.1 and S1.2 revealed an increased empirical T1E rate when applied to realistic data settings.
• This T1E inflation was more severe in the tail and persisted regardless of sample size.

Conclusions:

• Null simulation design significantly impacts the evaluation of statistical test performance.
• The "theoretical" null (S0) may provide an overly optimistic assessment of T1E control.
• Revised simulation practices and interpretation of T1E control are crucial for methods used in genetic association studies.