Bayesian Nonparametric Sensitivity Analysis of Multiple Test Procedures Under Dependence.

Area of Science:

• Statistics
• Statistical Methodology
• Multiple Testing Procedures

Background:

• Multiple testing procedures (MTPs) are crucial for controlling error rates when performing numerous statistical tests.
• Existing MTPs often require a priori selection, potentially leading to conservative results or unquantified uncertainty.
• Arbitrary dependence between p-values complicates MTP selection and performance evaluation.

Purpose of the Study:

• To develop a sensitivity analysis method for MTPs that accounts for uncertainty in MTP selection.
• To provide a framework for quantifying the uncertainty associated with threshold-based decisions in identifying significant discoveries.
• To reduce the conservativeness often associated with using a single MTP by considering a broader space of procedures.

Main Methods:

• The proposed method utilizes a Dirichlet process (DP) prior distribution to model the entire space of MTPs.
• It supports MTPs controlling either the family-wise error rate (FWER) or the false discovery rate (FDR).
• The method measures each p-value's probability of significance over the DP prior predictive distribution of all MTPs.

Main Results:

• The DP-MTP sensitivity analysis method was applied to over 28,000 p-values from a large-scale educational dataset.
• The analysis examined relationships between COVID-19 school closures and student academic/background variables.
• The method provides uncertainty quantification for MTP decisions, enhancing statistical inference.

Conclusions:

• The DP-MTP sensitivity analysis offers a robust approach to understanding uncertainty in multiple testing.
• This method allows for a more comprehensive evaluation of statistical significance across a range of MTPs.
• The approach is applicable to large datasets and complex research questions, as demonstrated in the educational context.