Boundary-aware geometric Mamba network for breast tumor segmentation in 3D MRI

Accurate breast tumor segmentation in Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) is important for reliable diagnosis and treatment planning. However, complex tumor morphology, diverse spatial structures, and indistinct boundaries remain challenging for existing segmentation methods. Convolutional neural network-based methods have limited ability to model long-range dependencies, while Transformer-based methods often introduce substantial computational overhead. Although Mamba-based approaches provide an efficient alternative, conventional scanning strategies may suffer from directional bias and insufficiently capture three-dimensional structural variations.

We propose a Boundary-Aware Geometric Mamba Network (BAGMNet) for three-dimensional breast tumor segmentation. BAGMNet establishes a structural representation learning framework that combines geometric modeling, multi-directional state-space propagation, and boundary-aware feature refinement. The Geometric Mamba Encoder constructs complementary spatial representations through multiple geometric transformations, while the Multi-directional Selective State Space module enhances three-dimensional dependency modeling and alleviates the directional bias of conventional Mamba scanning. The Gated State-space Fusion and Adaptive Skip Recalibration modules further integrate local structural details with global contextual information and enhance boundary-related representations during feature propagation.

Extensive experiments were conducted on the ISPY1 and Yunnan breast MRI datasets. BAGMNet achieved Dice scores of 70.34% and 79.14%, with average surface distances of 9.79 mm and 8.96 mm on the ISPY1 and Yunnan datasets, respectively. Compared with representative convolutional neural network-, Transformer-, and Mamba-based methods, BAGMNet achieved competitive performance in both regional segmentation accuracy and boundary localization.