Cell neighborhood topology directs rare cell population identification

Advances in single-cell and spatial technologies have transformed the dissection of cell composition and tissue architecture in complex biological systems. However, identifying rare cells critical to disease pathogenesis and biological processes remains difficult, as their low abundance often masks them among dominant cell populations. We introduce RareQ, a fast and scalable framework for rare cell detection by evaluating cliquishness of each cell's k-nearest neighborhood using single-cell omics data. Extensive benchmarking on diverse simulated and real datasets demonstrates that RareQ exceeds existing methods in accuracy, sensitivity, and efficiency. RareQ also excels at identifying both modality-specific and shared rare cells. Its versatility across various biological contexts enables the discovery of functionally distinct rare cells with unique molecular signatures in physiological and pathological context. RareQ's application to spatial transcriptomics data reveals anatomically distinct and clinically relevant rare cell populations. Together, RareQ offers an efficient approach for rare cell discovery, enhancing insights into tissue organization and disease mechanisms.