CDState Resolves Malignant Cell Heterogeneity from Bulk Tumor RNA-Sequencing Data

Intratumor transcriptional heterogeneity (ITTH), defined as the coexistence of diverse cell states within a single tumor, complicates cancer treatment and contributes to variable therapeutic responses. Although single-cell RNA sequencing (scRNA-seq) can resolve this complexity, its cost and technical demands limit large-scale use. Bulk RNA sequencing (bulk RNA-seq) provides a scalable alternative but requires computational methods to deconvolve bulk transcriptomes into distinct cell states. Existing supervised approaches rely on accurate reference data, which are lacking for many cancer types, while unsupervised methods are not tailored to capture heterogeneity within the malignant compartment. To address these limitations, we developed CDState, an unsupervised deconvolution method based on nonnegative matrix factorization with a sum-to-one constraint and a cosine-similarity-based optimization, which infers malignant cell states using bulk RNA-seq data. CDState demonstrated robustness using pseudobulk scRNA-seq datasets from five cancer types, outperforming existing unsupervised methods in estimating both state-specific gene expression and cell proportions. Applied to 33 cancer types from The Cancer Genome Atlas, CDState revealed recurrent gene programs, including epithelial-mesenchymal transition, MYC targets, and oxidative phosphorylation, as major contributors to malignant ITTH. The malignant state proportions were linked to clinical features, including patient survival and therapeutic response. Finally, mutations and copy number alterations in genes such as TP53, KRAS, PIK3CA, SOX2, and SATB1 were identified as potential genetic drivers of malignant cell ITTH across cancer types. This study demonstrates the utility of CDState for characterization of malignant cell states from bulk RNA-seq data, establishing a framework for investigating malignant cell ITTH in large-scale cancer atlases.