Deep learning for multiclass tumor cell detection in histopathology slides of hereditary diffuse gastric cancer

Hereditary diffuse gastric cancer (HDGC) is a rare condition where early tumor detection is challenging due to diffuse infiltration and tumor heterogeneity. Accurate identification of DGC cells is essential for understanding tumor behavior. This study aimed to develop deep learning models to automatically detect key tumor cell types-typical and atypical signet ring cells and non-signet ring tumor cells-in H&E-stained digital pathology slides from HDGC patients. Using a multi-center dataset of 350 whole-slide images and over 91,000 annotated cells from 43 patients, we trained nnU-Net models for cell detection and compared them to Faster R-CNN baselines. We also conducted a reader study with five pathologists to benchmark performance. nnU-Net outperformed both pathologist inter-observer agreement and Faster R-CNN, achieving an F₁ score of 0.49. It also matched human-level performance in estimating lesion size and cell type distributions, demonstrating its potential to support DGC tumor detection and analysis.