Resolution Tradeoffs in Modularity Clustering of Single Cell RNA-Seq Datasets

Modularity based clustering was introduced in the network literature for community detection and is now commonly applied to single cell RNA-seq (scRNAseq) datasets for cell type identification. Modularity clustering depends on a resolution parameter, which implicitly determines the number of clusters inferred, but no results exist describing clustering as a function of the resolution. For scRNAseq, an improperly chosen resolution parameter can lead to erroneous or missed cell types, corresponding to type I or II errors. In this work, we provide an explicit description of clustering as a function of the resolution parameter through the notion of a splitting resolution, the minimum resolution at which a graph or subgraph is split into multiple clusters. We show that the splitting resolution of a roughly disconnected subgraph is inversely proportional to the frequency of the subgraph within the graph. This result extends the resolution limit result of Fortunato and Barthelemy to the setting of a general resolution parameter value and establishes a significant limitation of cell type inference using modularity clustering. In the network literature, the starting point for modularity clustering is a graph, but in scRNAseq applications the starting point is a cell embedding used to form a graph. We study splitting resolutions of k-nearest neighbor graphs formed from cell embeddings distributed as a normal or a pair of isotropic normals. For such graphs, we derive formulas for the splitting resolution as a function of sample size, embedding dimension, and the covariance structure of the normals. We connect these results to scRNAseq by showing that for seven scRNAseq datasets, cell embeddings can be approximated by Gaussian mixtures. We use our results to provide specific examples of type I versus II error tradeoffs implicit in the choice of the resolution parameter.