Topology-aware unsupervised domain adaptation for curvilinear structure segmentation

Unsupervised domain adaptation (UDA) enables segmentation models to generalize to unlabeled clinical domains. However, existing methods primarily address appearance-level discrepancies, while neglecting topological mismatches in connectivity, branching, and loop structure-characteristics of curvilinear anatomies. This oversight frequently leads to fragmented, clinically unreliable segmentations of structures such as vascular networks. To bridge this gap, we propose TopoUDA, a framework that explicitly integrates topological reasoning at multiple stages of adaptation. First, we design a Curvilinear-Aware Discriminator (CAD), an adversarial critic that distinguishes anatomically plausible structures from fragmented predictions based on a compact topological fingerprint. To enable end-to-end training, we introduce a differentiable proxy network that provides a valid gradient path through this otherwise non-differentiable structural assessment. Second, we develop a Hierarchical Topological Refiner (HTR) that leverages persistent homology to identify stable structural components and refine spatially related fragments, generating structurally coherent pseudo-labels. Third, we introduce a Hybrid Trust Mechanism (HTM) that combines appearance- and topology-based reliability scores from dual discriminators to adaptively weight pseudo-label supervision, ensuring the model learns primarily from anatomically plausible predictions. Extensive validation across retinal fundus photography, Optical Coherence Tomography Angiography (OCTA), and electron microscopy demonstrates that TopoUDA significantly outperforms state-of-the-art methods in both overlap and topological metrics. Furthermore, qualitative generalization on the large-scale, unlabeled AI-READI cohort underscores TopoUDA's practical applicability to real-world clinical data where ground-truth annotations are unavailable.