SegFormer-based boundary-aware polyp segmentation with adaptive multi-branch fusion

Accurate definition of colorectal polyps for the prevention of cancer is challenging due to high variability in appearance and indefinite boundaries. In this work, we present a SegFormer-aided framework comprising a transformer encoder (SegFormer-B4) and a multi-branch fusion head in low-, high-, and all-level scales. Both tasks (edges and segmentation) are predicted by each branch and task-specific, learnable softmax weights then fuse branch logits. The framework is optimized under a composite objective comprising (i) deeply supervised soft dice similarity coefficient (DSC) loss for region overlap, (ii) Lovász-Hinge loss on the fusion head for compatibility for intersection over union (IoU) loss, and (iii) an edge-aware binary cross-entropy (BCE) term supervised by a Laplacian-based thin contour map. A lightweight pre-processing step removes textual overlays (OCR-inpainting-guided and specular highlights attenuation) for the removal of spurious cues. On three public datasets, the approach reaches excellent region and boundary precision: on Kvasir-Seg, mDice=0.946 and mIoU=0.899; on CVC-ClinicDB, mDice=0.961 and mIoU=0.926; and for ETIS (small low-contrast lesions), mDice=0.799 and mIoU=0.705, accompanied by top boundary-sensitive measures (S-measure, weighted [Formula: see text], and E-measure). The system is end-to-end, post-processing and any external prompting-free, and generalizes across datasets of varying difficulty, improving regional precision and boundary definition-two qualities desirable for clinical reliability.