Scalable nonparametric clustering with unified marker gene selection for single-cell RNA-seq data

Clustering is commonly used in single-cell RNA sequencing (scRNA-seq) to assess cellular heterogeneity, but standard methods often require user-specified heuristics and rely on post-selective differential expression analyses, which often lead to inflated false discovery rates. Here, we present NCLUSION: a nonparametric infinite mixture model that leverages Bayesian sparse priors to identify marker genes and cluster single-cell expression data simultaneously. NCLUSION uses a variational inference algorithm, which enables it to scale up to millions of cells. Through simulations and analyses of publicly available scRNA-seq studies, we demonstrate that NCLUSION (1) matches the performance of other state-of-the-art clustering techniques with significantly reduced runtime and (2) provides statistically robust and biologically relevant transcriptomic signatures for each of the clusters it identifies. Overall, NCLUSION represents a reliable hypothesis-generating tool for understanding patterns of expression variation present in single-cell populations.