Proximity adjusted centroid mapping for accurate detection of nuclei in dense 3D cell systems

In the past decade, deep learning algorithms have surpassed the performance of many conventional image segmentation pipelines. Powerful models are now available for segmenting cells and nuclei in diverse 2D image types, but segmentation in 3D cell systems remains challenging due to the high cell density, the heterogenous resolution and contrast across the image volume, and the difficulty in generating reliable and sufficient ground truth data for model training. Reasoning that most image processing applications rely on nuclear segmentation but do not necessarily require an accurate delineation of their shapes, we implemented Proximity Adjusted Centroid MAPping (PAC-MAP), a 3D U-net based method that predicts the position of nuclear centroids and their proximity to other nuclei. We show that our model outperforms existing methods, predominantly by boosting recall, especially in conditions of high cell density. When trained from scratch with limited expert annotations (30 images), PAC-MAP attained an average F1 score of 0.793 for nuclei centroid prediction in dense spheroids. When pretraining using weakly supervised bulk data (>2300 images) followed by finetuning with the available expert annotations, the average F1 score could be significantly improved to 0.816. We demonstrate the utility of our method for quantifying the absolute cell content of spheroids and comprehensively mapping the infiltration pattern of patient-derived glioblastoma cells in cerebral organoids.