A CNN-injected transformer network with lesion reconstruction for multi-view diabetic retinopathy grading

The type and quantity of lesions are critical determinants in the assessment of diabetic retinopathy (DR) grading. Since multi-view fundus images provide a broader field of view and capture more lesions, multi-view DR grading has garnered increasing attention in recent years. However, existing multi-view methods either only focus on fundus feature extraction, or only take the lesion map as a part of the input, failing to fully leverage the comprehensive lesion information. Moreover, the significant variation in lesion size and their scattered distribution present substantial challenges for effective information learning. To address these issues, this paper proposes a CNN-injected transformer network with Lesion Reconstruction for Multi-View DR grading (LRMVDR), which utilizes lesion maps twice to fully exploit lesion information. Specifically, to tackle the large-scale variations and widespread distribution of lesions, the lesion maps are concatenated with the fundus images and then input into the local and global branches for extracting hierarchical global-local features. Adapters are designed to inject CNN features into the Transformer between the two branches, significantly enhancing the integration of multi-scale global and local features. Additionally, a dedicated lesion reconstruction branch is employed to explicitly extract lesion features. These features are subsequently fused with those from the local branch via a wavelet enhancement module, enabling Interactive fusion of frequency domain information and spatial domain information. Extensive experiments on large public datasets demonstrate the effectiveness and competitiveness of the proposed method. Our code is available at https://github.com/HuYongting/LRMVDR.