DeMixNB: deconvolution of sparse-count RNA sequencing data for tumor cells using embedded negative binomial distributions

Estimating tumor-specific transcript proportions from mixed bulk samples has potential to inform novel biology. However, estimation accuracy using existing methods in sparse-count data such as microRNA-seq and spatial transcriptomics has yet to be established. We generate a mixed small RNA benchmark dataset to demonstrate analytical challenges. To resolve them, we develop DeMixNB, a semi-reference-based deconvolution model assuming a sum of negative binomial distributions. Applications to miRNA-seq from 885 patients with breast cancer and 4,709 spatial spots from lung cancer generates clinical and mechanistic insights into tumor cell plasticity. This supports the important utility of DeMixNB to investigate cancer RNomes.