RAG: a regularized adaptive graph-based method for rare-cell identification from single-cell expression data

Rare-cell identification is essential for dissecting disease mechanisms and developmental programs. Existing methods mostly rely on fixed-size neighbourhood graphs to separate rare-cell populations in single-cell expression data, which may embed rare cells into dominant clusters under varying sampling densities. This paper proposes the RAG method for identifying rare cells based on regularized adaptive graphs, which can better separate rare cells. Specifically, the regularized adaptive graph is constructed by estimating cell-specific radii from Euclidean-cosine hybrid dissimilarity to constrain effective neighbours and stabilize the adjacency, and then, assigning locally scaled hybrid affinities to make affinity magnitudes comparable across density-varying regions. Across 10 real single-cell RNA sequencing datasets, RAG overall outperformed six state-of-the-art methods, improving precision, F1 score, and rare-type coverage rate over the second-ranked baseline by 42%, 26%, and 35%, respectively. A case study on colorectal tumour tissue shows that RAG is more accurate in recovering annotated rare-cell populations and separating the substructure from the major population than the other evaluated methods. Further analyses on mouse airway epithelium and two pancreas datasets showed that about half of RAG-resolved small clusters corresponded to known annotated populations or marker-supported subpopulations. The source code is available at https://github.com/wangxingsu/RAG.