Improving generalization capability of deep learning-based nuclei instance segmentation by non-deterministic train...

Area of Science:

• Digital Pathology
• Computational Biology
• Medical Image Analysis

Background:

• Digital pathology enables automated whole slide imaging, driving demand for computerized image analysis.
• Nuclei instance segmentation is critical for clinical and research applications in histopathology.
• Deep learning (DL) models excel at nuclei segmentation but struggle with generalization to new datasets.

Purpose of the Study:

• To develop a novel method that enhances the generalization capability of DL-based nuclei instance segmentation.
• To improve the robustness of DL models when applied to unseen histopathological datasets.

Main Methods:

• Utilized a state-of-the-art DL model as a baseline.
• Incorporated non-deterministic, train-time stain normalization.
• Implemented deterministic, test-time stain normalization.
• Employed ensembling techniques to boost segmentation performance.
• Trained the model on a single dataset and evaluated on seven diverse test datasets.

Main Results:

• The proposed method demonstrated superior performance compared to the baseline model.
• Achieved up to 4.9% improvement in Dice score.
• Showed up to 5.4% improvement in aggregated Jaccard index.
• Reported up to 5.9% improvement in panoptic quality score.

Conclusions:

• The novel method significantly enhances the generalization of DL-based nuclei segmentation.
• Stain normalization and ensembling are effective strategies for improving model robustness.
• The approach offers improved accuracy for nuclei instance segmentation in digital pathology.