scGHSOM: A Hierarchical Framework for Single-Cell Data Clustering and Visualization

Cell states' complexity and heterogeneity pose significant challenges in uncovering biological patterns in high-dimensional single-cell data. To address this, we developed scGHSOM, an enhanced framework based on the Growing Hierarchical Self-Organizing Map (GHSOM), for hierarchical clustering and visualization of high-dimensional datasets such as Mass Cytometry by Time-Of-Flight (CyTOF) and single-cell RNA sequencing. scGHSOM organizes data hierarchically, expanding clusters to satisfy within- and between-cluster variation thresholds. We propose a novel Significant Attributes Identification algorithm within the scGHSOM framework to identify features that minimize intra-cluster variation while maximizing inter-cluster variation, enabling targeted data analysis. To enhance interpretability, scGHSOM introduces two visualization tools: the Cluster Feature Map, which highlights feature distributions across hierarchical clusters, and the Cluster Distribution Map, which visualizes leaf clusters as circles sized by data volume and filled with different colors to represent features such as cell types or other attributes. Performance evaluation on three CyTOF datasets demonstrates that scGHSOM is compatible with state-of-the-art methods. Specifically, it achieves the best CH index in two of the three datasets. Furthermore, the proposed visualization tools significantly improve clarity and efficiency in interpreting scGHSOM results, effectively revealing clustering patterns and features.