Uncovering smooth structures in single-cell data with neighbor embeddings guided by...

Single-cell sequencing enables detailed study of cell-state transitions, but extracting smooth, low-dimensional structures from noisy, high-dimensional data remains challenging. Neighbor embedding (NE) algorithms, such as t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP), are widely used to embed high-dimensional single-cell data into low dimensions, but they often introduce distortions that can lead to misleading interpretations. To address these challenges, we build on the predictability-computability-stability (PCS) framework for reliable and reproducible data-driven discoveries. First, we systematically evaluate popular NE algorithms through empirical and theoretical analyses, revealing their key limitations such as algorithmic artifacts and instability. We then introduce NESS, a principled and interpretable machine learning approach that improves NE representations by leveraging algorithmic stability, enabling more robust inference of smooth biological structures from single-cell data. Finally, we apply NESS to multiple datasets, including studies of pluripotent stem cell differentiation, organoid development, and diverse tissue-specific lineages. Across these settings, NESS consistently yields biologically meaningful insights.