Related Experiment Video
Updated: Aug 15, 2026
Three Differential Expression Analysis Methods for RNA Sequencing: limma, EdgeR, DESeq2
Published on: September 18, 2021
A note on using permutation-based false discovery rate estimates to compare different analysis methods for microarray
Yang Xie1, Wei Pan, Arkady B Khodursky
1Division of Biostatistics, School of Public Health, University of Minnesota Minneapolis, MN 55455, USA. yangxie@biostat.umn.ed
Motivation:
False discovery rate (FDR) is defined as the expected percentage of false positives among all the claimed positives. In practice, with the true FDR unknown, an estimated FDR can serve as a criterion to evaluate the performance of various statistical methods under the condition that the estimated FDR approximates the true FDR well, or at least, it does not improperly favor or disfavor any particular method. Permutation methods have become popular to estimate FDR in genomic studies. The purpose of this paper is 2-fold. First, we investigate theoretically and empirically whether the standard permutation-based FDR estimator is biased, and if so, whether the bias inappropriately favors or disfavors any method. Second, we propose a simple modification of the standard permutation to yield a better FDR estimator, which can in turn serve as a more fair criterion to evaluate various statistical methods.
Results:
Both simulated and real data examples are used for illustration and comparison. Three commonly used test statistics, the sample mean, SAM statistic and Student's t-statistic, are considered. The results show that the standard permutation method overestimates FDR. The overestimation is the most severe for the sample mean statistic while the least for the t-statistic with the SAM-statistic lying between the two extremes, suggesting that one has to be cautious when using the standard permutation-based FDR estimates to evaluate various statistical methods. In addition, our proposed FDR estimation method is simple and outperforms the standard method.
Insights
The standard permutation method often overestimates the false discovery rate (FDR), potentially biasing evaluations of statistical methods. A modified permutation approach provides a more accurate and fairer FDR estimation for genomic studies.
Area of Science:
- Genomic studies
- Statistical method evaluation
- Bioinformatics
Background:
- False discovery rate (FDR) is crucial for evaluating statistical methods, especially in genomic research.
- Permutation methods are commonly used to estimate FDR but may exhibit bias.
- Accurate FDR estimation is essential for fair comparison of statistical approaches.
Purpose of the Study:
- To investigate the bias of standard permutation-based FDR estimators.
- To determine if this bias unfairly favors or disfavors specific statistical methods.
- To propose a modified permutation method for improved FDR estimation and fairer method evaluation.
Main Methods:
- Theoretical and empirical investigation of standard permutation FDR estimators.
- Simulation and real data analysis.
- Comparison of sample mean, SAM statistic, and Student's t-statistic performance.
- Development and validation of a modified permutation-based FDR estimation method.
Main Results:
- The standard permutation method systematically overestimates FDR across tested statistics.
- Bias severity varied, being most pronounced for the sample mean and least for the t-statistic.
- The proposed FDR estimation method demonstrated superior performance compared to the standard approach.
Conclusions:
- Caution is advised when using standard permutation-based FDR estimates for method comparison due to overestimation.
- The proposed modified permutation method offers a more accurate and equitable FDR estimation.
- This improved estimation facilitates fairer evaluation of statistical methods in genomic research.
