AV-NeXt: a topology-aware framework with interaction modeling and competitive gating for retinal artery/vein...

Automated segmentation and classification of retinal arteries and veins (A/V) are pivotal for the precise morphological characterization required in the early diagnosis of systemic cardiovascular and ophthalmic pathologies. However, existing deep learning paradigms exhibit limitations in resolving complex structural entanglement at A/V crossings, frequently resulting in vascular disconnection and classification ambiguity due to the insufficient modeling of local artery-vein interactions. To address these challenges, AV-NeXt, a topology-aware framework, is proposed to improve crossing-region recognition by integrating anatomically motivated A/V interaction modeling into deep feature representations. Constructed upon a ConvNeXt backbone, the framework incorporates two pivotal components: (1) an Interaction-conditioned Crossing Head, which exploits feature-level co-activation cues between artery and vein streams to assist crossing prediction; and (2) a Competitive Gating Mechanism, designed to dynamically reduce crossing-induced classification ambiguity through soft gating. Extensive experiments on Fundus-AVSeg demonstrated that AV-NeXt achieved the best overall performance among the compared methods, with an mDice of 76.72%, an mIoU of 63.64%, and a Crossing IoU of 44.85%. Paired statistical analysis further supported the reliability of the crossing-region gains, while zero-shot evaluation on RITE showed competitive crossing recognition under domain shift, with a Crossing IoU of 28.52%.