Improving Colorectal Tumor Bud Segmentation with Deformable Multi-Scale Transformers

Tumor budding (TB) is a key histopathological marker in colorectal cancer (CRC), closely linked to disease progression and prognosis. Deep learning has greatly advanced histological image analysis by enabling robust and efficient pattern recognition. In this study, we developed a SegFormer-based TB segmentation pipeline using pretrained backbones and evaluated two new decoder variants: Feature Pyramid Network (FPN) and a Deformable FPN (DFPN) layer to better capture small and irregular cellular structures. The dataset includes 8606 training images, 201 validation images, and 2036 test images derived from 86 H&E-stained WSIs, plus 176 finely re-annotated test set images. We evaluated the performance of pre-trained encoder models using the MiT-B0 and MiT-B3 backbones. The best-performing models are then further enhanced by replacing their decoder components with either an FPN or a DFPN, aiming to boost segmentation performance metrics. Using the best encoder backbone in SegFormer, we achieved instance dice scores of 0.7847 on test images and 0.8864 on the finely re-annotated test set. Integrating a Deformable FPN (DFPN) decoder substantially improved instance-level Dice, particularly for small and irregular tumor buds, while pixel-level overlap metrics changed more modestly. This demonstrates that decoder-level innovations are critical for enhancing Vision Transformer-based segmentation in pathology. This study highlights the potential of decoder innovations to improve ViT-based segmentation in surgical pathology, with future work focused on expanding annotated datasets and enhancing model adaptability for real-world use.