HyperDiffuseNet: A Deep Hyperbolic Manifold Learning Method for Dimensionality Reduction in Spatial Transcriptomics

Spatial transcriptomics (ST) reveals tissue organization but presents analytical challenges due to high dimensionality and complex spatial-hierarchical structures, which are often distorted by Euclidean-based dimensionality reduction methods. Here, we introduce HyperDiffuseNet, a deep geometric learning framework designed for ST data representation. HyperDiffuseNet utilizes a variational autoencoder with a hyperbolic latent space to effectively capture hierarchical relationships. It integrates spatial context by first employing graph convolutional networks on the spatial graph to learn multi-scale dependencies, which inform the computation of a diffusion matrix. This graph-derived diffusion information is then efficiently incorporated into the hyperbolic embeddings via linear mixing in the ambient Minkowski space. The model uses negative binomial reconstruction loss and is optimized with a composite objective function balancing reconstruction fidelity, Kullback-Leibler divergence regularization, attention-weighted spatial regularization, diffusion consistency, and local structure preservation. Empirical evaluations on multiple ST datasets demonstrate that HyperDiffuseNet achieves competitive clustering performance. The hyperbolic embedding approach shows notable improvements in Silhouette coefficient and adjusted rand index metrics across most tested datasets, while maintaining comparable performance in structure preservation.