LSP-DETR: Efficient and scalable nuclei segmentation in whole-slide images

Precise and scalable instance segmentation of cell nuclei is a fundamental prerequisite for computational pathology, yet gigapixel whole-slide images (WSIs) pose significant computational challenges. While patch-based processing is standard during training, existing methods are often limited to small tile sizes during inference due to architectural bottlenecks or reliance on computationally expensive post-processing steps for instance separation. We introduce a faster, scalable, and end-to-end framework capable of processing large-scale image tiles while accurately modeling biologically realistic overlapping nuclei.

We propose LSP-DETR (Local Star Polygon DEtection TRansformer). The model represents nuclei as star-convex polygons and employs a lightweight transformer with linear complexity, enabling the processing of high-resolution images in a single forward pass. A novel radial distance loss accommodates annotation uncertainty, allowing the segmentation of overlapping nuclei to emerge naturally without explicit overlap labels.

LSP-DETR achieves state-of-the-art efficiency, with an inference time of 0.45 s/mm² representing a 3.2× speedup over StarDist, the next-fastest method. On PanNuke, the model achieves competitive accuracy (67.5 bPQ), while yielding an F₁-score of 0.964 in polygon overlap when evaluated against consensus annotations from two expert pathologists. Furthermore, it outperforms larger models such as LKCell in terms of generalization robustness, reaching an F₁-score of 85.0 on MoNuSeg.

LSP-DETR bridges the gap between high-fidelity segmentation and practical clinical requirements by eliminating heuristic post-processing. By providing a scalable, linear-complexity solution that naturally handles overlaps between nuclei, this framework sets a new direction for efficient high-throughput WSI analysis in digital pathology.