Fuzzy rough set loss for deep learning-based precise medical image segmentation

Accurate segmentation of medical images is crucial for diagnosis and treatment planning, yet it remains challenging due to ambiguous lesion boundaries, class imbalance, and complex anatomical structures. We propose a novel Fuzzy Rough Set-inspired (FRS) loss function that addresses these challenges by integrating pixels' fuzzy similarity relations with a boundary uncertainty model in a convex combination method. To obtain the boundary uncertainty model, the fuzzy lower and upper approximations of a set of pixels and membership weights are utilized. The FRS loss function enhances boundary sensitivity and handles prediction uncertainty through its dual components: a fuzzy similarity term that captures gradual transitions at lesion boundaries, and boundary uncertainty model that deals with uncertainty and mitigates class imbalance. Extensive experiments across five diverse medical imaging datasets - breast ultrasound, gastrointestinal polyps, brain Magnetic Resonance Imaging (MRI), chest Computed Tomography (CT), and skin lesions - demonstrate the effectiveness of our approach. The FRS loss achieves superior segmentation performance with an average improvement of 2.1% in Dice score compared to the best baseline method, while demonstrating statistically significant improvements across all evaluated metrics (p < 0.001). The FRS loss shows its robustness to moderate class imbalance while maintaining computational efficiency (mean inference time 0.075-0.12 s per image, 4.5 MB memory). These results suggest that the FRS loss function provides a robust and interpretable framework for precise medical image segmentation, particularly in cases with ambiguous boundaries and moderate imbalance. Code: https://github.com/MohsinFurkh/Fuzzy-Rough-Set-Loss.