NeST: nested hierarchical structure identification in spatial transcriptomic data

Spatial gene expression in tissue is characterized by regions in which particular genes are enriched or depleted. Frequently, these regions contain nested inside them subregions with distinct expression patterns. Segmentation methods in spatial transcriptomic (ST) data extract disjoint regions maximizing similarity over the greatest number of genes, typically on a particular spatial scale, thus lacking the ability to find region-within-region structure. We present NeST, which extracts spatial structure through coexpression hotspots-regions exhibiting localized spatial coexpression of some set of genes. Coexpression hotspots identify structure on any spatial scale, over any possible subset of genes, and are highly explainable. NeST also performs spatial analysis of cell-cell interactions via ligand-receptor, identifying active areas de novo without restriction of cell type or other groupings, in both two and three dimensions. Through application on ST datasets of varying type and resolution, we demonstrate the ability of NeST to reveal a new level of biological structure.