scGEN: Leveraging an Adaptive Weighting Strategy for Challenging Cell Clustering in Single-Cell RNA-seq

Recent advancements in single-cell RNA sequencing (scRNA-seq) have greatly enhanced our ability to dissect cellular heterogeneity. However, unsupervised clustering often struggles to identify transitional or developmental boundary cells, as existing methods rely on highly variable genes without considering expression levels, thereby overlooking subtle but crucial signals. To address this challenge, we developed a single-cell gene-aware embedded network (scGEN), which captures complex intercellular relationships. scGEN employs adaptive feature weighting and iterative fine-tuning to prioritize ambiguous or transitional cells with overlapping transcriptional profiles. Evaluation across eight distinct scRNA-seq datasets demonstrated that scGEN consistently outperformed nine leading clustering approaches. Additionally, scGEN generated refined assignments for approximately 10% of cells with ambiguous transcriptional profiles and revealed biologically meaningful differences, providing a more detailed view of cellular heterogeneity in the human fetal pituitary. Cell-cell communication analysis identified developmental signaling networks associated with the refined annotations, including a putative Pre.Gonado-like population detected at 7 weeks with prominent involvement in BMP, NOTCH, and extracellular matrix remodeling pathways. Furthermore, histological analysis of 6-8-week embryos provided orthogonal tissue-level support for early gonadotrope-lineage states and SOX2-GATA2-NR5A1 bridging populations. Together, these results highlight scGEN's ability to enhance cell-type assignments and detect subtle biological differences.