Gabor-modulated depth separable convolution for retinal vessel segmentation in fundus images

In diabetic retinopathy, precise segmentation of retinal vessels is essential for accurate diagnosis and effective disease management. This task is particularly challenging due to the varying sizes of vessels, their bifurcations, and the presence of highly curved segments. While numerous automated segmentation techniques have demonstrated strong performance, deep neural networks have struggled to effectively model the geometric transformations of retinal vessels without extensive training datasets. Moreover, the inconsistent quality of fundus photographs often results in less than satisfactory accuracy in vessel structure detection.

To tackle these challenges, we propose a Gabor-modulated depth separable convolution UNet model that offers flexibility in capturing visual properties such as vessels' spatial frequency and orientation. Gabor filters are highly sensitive to different orientations, allowing them to detect edges and lines at specific angles. Therefore, the proposed model can effectively recognize vessels with varying widths and orientations. We have reinforced the network's learning capability by integrating Gabor convolution with Depth separable convolution.

Extensive experiments conducted on the DRIVE, STARE, and CHASE_DB1 datasets demonstrate the proposed model's effectiveness, even with limited training data. The integration of Gabor filters within the UNet framework significantly improves the segmentation performance, particularly in capturing vessels with varying orientations and spatial dimensions, even on noisy and progressed DR images. Our approach achieves superior performance on all metrics compared to the other deep learning models, confirming the robustness and flexibility of the proposed architecture.

The Gabor-modulated depth separable convolution UNet model effectively addresses the challenges in retinal vessel segmentation by leveraging the orientation-sensitivity of Gabor filters and the efficiency of depth separable convolutions. The model exhibits excellent segmentation performance across multiple datasets and shows its potential to enhance diagnostic accuracy in diabetic retinopathy, even when data availability is limited. Furthermore, its lightweight architecture facilitates implementation in resource-constrained environments, making it a feasible option for various clinical applications.