NuSeT: A deep learning tool for reliably separating and analyzing crowded cells

Area of Science:

• Cell Biology
• Bioimaging
• Computational Biology

Background:

• Accurate cell nucleus segmentation is crucial for biological research and clinical diagnostics.
• Standard deep learning models struggle with segmenting low-contrast, overlapping, and densely packed nuclei.

Purpose of the Study:

• To develop and validate a novel deep learning tool, the Nuclear Segmentation Tool (NuSeT), for accurate cell nucleus segmentation.
• To address common challenges in nuclear segmentation, including signal variability, shape variations, limited training data, and preparation artifacts.

Main Methods:

• Developed NuSeT using a hybrid deep learning network combining U-Net and Region Proposal Networks (RPN).
• Incorporated a watershed step for precise boundary delineation.
• Employed foreground normalization and training with synthetic data containing artifacts to enhance robustness.

Main Results:

• NuSeT demonstrated superior performance in detecting and delineating nuclear boundaries in diverse 2D and 3D fluorescence microscopy images.
• Achieved improved nuclear detection and reduced false positives compared to existing segmentation models.
• Consistently generated accurate segmentation masks and resolved touching nuclei boundaries.

Conclusions:

• NuSeT effectively overcomes limitations of standard models for challenging nuclear segmentation tasks.
• The tool provides accurate segmentation masks and boundary assignments for touching nuclei.
• NuSeT offers a robust solution for various microscopy imaging data, advancing cell nucleus analysis.