Skeleton-guided sparse anchors for rotated instance segmentation in cell microscopy

Accurate instance segmentation of clustered cells in microscopy images remains a major bottleneck, as traditional methods often break down when objects of varying sizes and shapes touch or overlap. We introduce A2B-IS, a novel one-stage framework that represents each cell with a pixel-level mask and a rotated bounding box, specifically designed to improve segmentation quality in densely packed regions.

A2B-IS decouples mask and box prediction into parallel branches to simplify the pipeline and reduce error propagation. We incorporate a Gaussian skeleton map that (1) guides anchor placement to focus computations on likely cell centers and suppress background noise, and (2) corrects box predictions near instance boundaries to prevent merged or fragmented detections. To enrich feature representations, we embed an Atrous Attention Block that captures fine-grained, multiscale details at high resolution. Finally, a semi-supervised learning strategy leverages unlabeled images alongside annotated data to further boost model robustness and generalization. The code and dataset are available at https://github.com/wangjuncongyu/A2B-Net.

On two large-scale cell microscopy datasets, A2B-IS consistently outperformed leading one- and two-stage segmentation approaches. Compared to baseline models, it achieved higher average precision and recall, with particularly strong gains in densely clustered regions and for small or irregularly shaped cells.

By combining pixel-level masks, rotated boxes, skeleton-guided anchors, attention-based feature extraction, and semi-supervised training, A2B-IS delivers substantial improvements in challenging microscopy segmentation tasks. This advance paves the way for more reliable automated analysis of cell populations without extensive per-image calibration.