A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and...

Area of Science:

• Computational Pathology
• Medical Image Analysis
• Deep Learning

Background:

• Accurate cell segmentation in whole slide images (WSIs) is crucial for quantitative analysis in computational pathology.
• Conventional methods face challenges due to the complexity and scale of WSIs.

Purpose of the Study:

• To develop a novel hybrid deep learning framework for robust and precise cell segmentation in WSIs.
• To integrate YOLOv11, StarDist, and Segment Anything Model v2 (SAM2) for enhanced performance.

Main Methods:

• A hybrid framework combining YOLOv11 for object detection, StarDist for precise boundary modeling, and SAM2 for prompt-guided segmentation.
• Utilized YOLOv11 outputs as prompts or filters for SAM2, and StarDist for geometric accuracy in dense regions.
• Evaluated on a WSI dataset with high-resolution cell-level masks.

Main Results:

• The proposed hybrid method significantly outperformed individual baseline models in cell segmentation.
• Achieved enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types.
• Quantitative evaluations showed superior performance using Dice coefficient, IoU, F1-score, precision, and recall.

Conclusions:

• The integrated framework offers a scalable and modular solution for automated histopathological image analysis.
• Hybrid deep learning approaches can overcome limitations of individual models for complex WSI segmentation.
• This method advances quantitative analysis in computational pathology through improved cell segmentation accuracy.