spOT-NMF: gene expression deconvolution via optimal transport with proximity-aware niche mapping for spatial transcriptomics

Spatial transcriptomics enables direct profiling of cellular organization within intact tissues, yet two challenges limit its full exploitation: accurate reference-free deconvolution of mixed expression signals (including species-admixed models such as xenografts), and lack of tools that can translate per-spot program usage values into spatial niches. We developed spOT-NMF, an optimal transport-based non-negative matrix factorization method that replaces standard reconstruction loss with the Wasserstein distance to disentangle mixed signals without requiring matched single-cell atlases. While the generalized OT-NMF factorization is applicable to any expression matrix, a proximity-based niche inference module further translates program co-usage from spatial transcriptomes into directional co-occurrence networks that define tissue compartments and their boundaries. Benchmarking across five gold-standard datasets spanning sub-cellular to multi-cellular resolutions, multiple platforms, and two-species admixture scenarios, spOT-NMF ranks first among reference-free methods, with particular advantage under the signal imbalance conditions characteristic of xenografts. We show that reference-free deconvolution recovers cell populations absent from matched single-cell atlases, highlighting the value of unsupervised analysis. Applied to glioblastoma xenografts and human colorectal cancer, spOT-NMF reveals spatially restricted interferon signaling, invasion routes, and CAF-tumor boundaries. spOT-NMF is available as an open-source Python package with GPU, CPU, and HPC support.