Mamba-enhanced codebook learning with anatomical constraints for liver and tumor segmentation in 3D CT volumes

Precise delineation of the liver and its tumors in 3D CT scans plays a vital role in clinical diagnosis and therapeutic planning. However, current deep learning approaches frequently struggle with tumor heterogeneity, varying lesion sizes, and ambiguous boundaries, which can limit their effectiveness. To address these issues, we propose an end-to-end hierarchical network that effectively integrates multi-scale context modeling, global relational learning, and structured feature representation. First, a multi-scale texture encoder is designed to capture tumor characteristics across different spatial resolutions. To model long-range dependencies across slices, we introduce a global relational representation module built upon the emerging Mamba architecture, enabling efficient and directional context aggregation in 3D volumes. Second, to enhance feature compactness and stability, we propose a learnable codebook module that quantizes high-dimensional features into a finite set of semantic prototypes, promoting discriminative representation learning while suppressing redundancy. Furthermore, anatomical prior knowledge-specifically, the spatial constraint that tumors must reside within the liver-is incorporated via an inclusion loss, which explicitly regularizes the segmentation outputs. Comprehensive experiments on the public LiTS dataset show that our method attains state-of-the-art results, surpassing existing methods in Dice score, volumetric overlap error (VOE), and boundary metrics (ASD and 95HD). Ablation analyses confirm the individual contribution of each module, demonstrating the architecture's effectiveness for accurate and reliable liver and tumor segmentation.