DTARNU-Net: Dense Tiered Attention Residual Nested U-Net for CT Liver Tumor Segmentation

Liver tumor segmentation is a significant task in clinical imaging that involves detecting liver tumors and distinguishing them from the surrounding liver tissue in CT scans. Precision segmentation performs important roles in the initial detection of liver cancer, treatment planning, and monitoring disease development, which also supports doctors, facilitating surgeries and radiation therapy more efficiently. Meanwhile, clinical imaging and segmentation algorithms have been enhanced over the years. The currently prevailing state-of-the-art methods still face multiple difficulties, though, in obtaining precision and reliability in their outcomes. Tumors with irregular shapes, variable sizes, and densities similar to those of surrounding tissues often lead to segmentation inaccuracies and potential misdiagnoses. In this work, we tackle these challenges by developing an advanced process for precise liver tumor segmentation by utilizing CT images from the LiTS dataset. The proposed DTARNU-Net was developed, trained, validated, and evaluated exclusively using the Liver Tumor Segmentation (LiTS) benchmark dataset. No experiments were conducted on the 3D-IRCADbI dataset in this study. All quantitative and qualitative results presented in the manuscript correspond to the LiTS dataset. The LiTS dataset contains contrast-enhanced abdominal CT scans with expert-annotated liver and tumor masks. The proposed model was evaluated using patient-level training, validation, and testing partitions (9:2:2 ratio), and all experiments were independently repeated five times. Statistical significance was assessed using paired Student's t-test (p < 0.05), and the results confirmed that the performance improvements over competing methods are statistically significant. We introduce a novel three-level pre-processing approach that significantly enhances image quality through histogram equalization, noise removal, smoothing, and sharpening. Our approach is embodied in the Dense Tiered Attention Residual Nested U-Net (DTARNU-Net), a sophisticated model combining the strengths of a Siamese network and a nested U-Net architecture. This model incorporates the ACON-ReLU residual convolution block (A-R), which improves recognition accuracy in regions with subtle changes, reducing missed detection. The presented method enhances trait collaboration and spatial data by utilizing the Brownian Motion-based Butterfly Optimization Algorithm (BM-BOA). This algorithm efficiently integrates low-level trait details with high-level semantic data. The Dense Tiered Attention Residual Module (DTSRM) additionally improves these traits to obtain more precise segmentation. The model achieved segmentation robustness of 96.78% for liver segmentation and 97.00% for liver tumor segmentation on the LiTS dataset. These outcomes indicate that the presented method performs better than the prevailing state-of-the-art methods and has the capability to help computer-assisted detection and treatment by furnishing more precise and reliable liver tumor segmentation.