Topology-guided hard example mining for cell detection

Automatic cell detection is a key task in digital pathology, where manual counting remains impractical due to its time-consuming nature and susceptibility to variability and error. Current deep learning approaches still have difficulty achieving accurate detection, particularly in images with crowded cell distributions. In such settings, capturing the global organization of cells within tissue becomes critical; however, the topological structure underlying cell arrangements is often ignored by existing models. To address these limitations, we propose topology-guided hard example mining (TG-HEM), a novel training strategy that incorporates topological constraints into the training of cell detection networks through loss reweighting. In contrast to pixel-centric HEM techniques, TG-HEM identifies challenging regions by quantifying topological discrepancies between ground truth and predicted cell distributions using persistent homology, rather than relying solely on local pixel-wise errors. By assigning higher importance to regions with larger topological inconsistencies and further emphasizing hard-to-learn pixels within these regions, the proposed approach guides backpropagation toward regions that reflect structural differences in cell distributions. This formulation enables HEM at both the region and pixel levels, allowing the network to better capture higher-order organization in crowded cell distributions. We evaluate TG-HEM across multiple network architectures and on two datasets: the publicly available BRCA-M2C dataset and our in-house KUCell dataset, which we release as part of this work. The experimental results show that the proposed TG-HEM approach consistently improves both cell counting and localization accuracy compared to existing HEM strategies. These improvements are achieved without introducing additional model complexity or inference-time overhead, and with only negligible impact on training time. The KUCell dataset is available at https://mysite.ku.edu.tr/cgunduz/downloads/KUCell, and the codes are available at https://github.com/caki35/TGHEM.