Optimizing deep learning-based segmentation of densely packed cells using cell surface markers.

Area of Science:

• Computational pathology
• Bioimage analysis
• Machine learning for biological imaging

Background:

• Accurate cell segmentation is crucial for spatial molecular profiling.
• Identifying and quantifying cells in dense, inflamed tissues remains a significant challenge.

Purpose of the Study:

• To evaluate deep learning models for cell segmentation in complex tissue images.
• To enhance cell segmentation performance through model training and parameter tuning.

Main Methods:

• Assessed 18 deep learning cell segmentation models on immunofluorescence images of skin during human herpes simplex virus (HSV) infection.
• Further trained eight models using over 10,000 instances from the target image set.
• Optimized parameters of the top-performing model for improved accuracy.

Main Results:

• The best model achieved a mean Average Precision (mAP) of 0.516 before fine-tuning.
• Post-training, the Cellpose cyto model reached an mAP of 0.694.
• Further parameter tuning increased the mAP to 0.711, outperforming initial benchmarks.

Conclusions:

• Model selection and targeted training are key to improving cell segmentation performance.
• The fine-tuned model demonstrates performance comparable to human experts.
• Moderate signal-to-noise ratio in images impacted final model accuracy.