MammoDenseSegNet: A Context-Aware Deep Learning Model for Dense Tissue Segmentation in Digital Mammograms

Breast density is a breast cancer risk factor. The accurate quantification of breast density requires reliable segmentation of dense tissue in mammograms, but it is a challenging task due to large variations in tissue appearance across hospitals and imaging devices. We propose MammoDenseSegNet, a new deep encoder-decoder convolutional neural network designed to enhance segmentation performance through two complementary modules: a) Adaptive dual attention module, which captures long-range spatial and channel interdependencies to provide focused attention on relevant dense tissue areas regardless of their location; and b) Multi kernel receptive field module, which enlarges the network's receptive field at the bottleneck layer to aggregate multi-scale contextual features. Additionally, a multi-scale dice loss with deep supervision guides learning across decoder levels to improve robustness. We evaluated MammoDenseSegNet on two public digital mammogram datasets (VinDR-Mammo and EMBED) and one private dataset, spanning a variety of breast densities and imaging artifacts in a total of 1499 images from 606 women. Statistical analysis was done using generalized linear models accounting for correlation among images from the same women and adjusting for potential confounders (proc genmod, proc mixed, SAS v.9.4, SAS Institute, Cary, NC). MammoDenseSegNet demonstrated consistently high performance across various conditions (with Recall ranging from 0.64 to 0.90 and Dice from 0.63 to 0.91) and significantly (p < 0.001) outperformed the publicly available state-of-the-art algorithm based on the VGG16 (with Recall from 0.04 to 0.91 and Dice from 0.06 to 0.82 across the same conditions). The improvement was largest for low-density tissue, where the baseline algorithm practically fails (with the mean Recall of 0.14 and Dice of 0.16) while MammoDenseSegNet remained clinically useful (with the mean Recall of 0.66 and Dice of 0.63).