Rotation testing in gene set enrichment analysis for small direct comparison experiments

Gene Set Enrichment Analysis (GSEA) is a method for analysing gene expression data with a focus on a priori defined gene sets. The permutation test generally used in GSEA for testing the significance of gene set enrichment involves permutation of a phenotype vector and is developed for data from an indirect comparison design, i.e. unpaired data. In some studies the samples representing two phenotypes are paired, e.g. samples taken from a patient before and after treatment, or if samples representing two phenotypes are hybridised to the same two-channel array (direct comparison design). In this paper we will focus on data from direct comparison experiments, but the methods can be applied to paired data in general. For these types of data, a standard permutation test for paired data that randomly re-signs samples can be used. However, if the sample size is very small, which is often the case for a direct comparison design, a permutation test will give very imprecise estimates of the p-values. Here we propose using a rotation test rather than a permutation test for estimation of significance in GSEA of direct comparison data with a limited number of samples. Our proposed rotation test makes GSEA applicable to direct comparison data with few samples, by depending on rotations of the data instead of permutations. The rotation test is a generalisation of the permutation test, and can in addition be used on indirect comparison data and for testing significance of other types of test statistics outside the GSEA framework.