GiTNet: A graph-based trajectory-informed network for gaze-supervised medical image segmentation

Creating fully annotated labels for medical image segmentation is both time-consuming and costly, underscoring the need for efficient annotation schemes to alleviate the workload. Eye tracking offers an economical solution that can be seamlessly integrated into the clinicians' workflow, providing relevant supervision for tasks. However, inaccuracies and ambiguity in gaze provide erroneous supervision for segmentation, and dynamic trajectories that encode rich temporal and structural context are not fully and effectively exploited, leading to underutilization of the semantic information embedded in gaze. This hinders the effectiveness of gaze-supervision and impairs the model's ability to accurately delineate organ and lesion boundaries in ambiguous regions. To address these challenges, we propose the graph-based trajectory-informed network (GiTNet), which integrates static fixations with dynamic trajectories to comprehensively model complex anatomical relationships and potential lesion areas and constraint graph topology to strengthen the model's ability to focus on anatomy- and lesion-related regions through the trajectory relational alignment (TRA). Additionally, we introduce neighbor-aware pseudo supervision (NAP), which incorporates the semantic information from neighboring nodes in the graph to reduce noise and uncertainty in gaze. Moreover, graph representational consistency (GRC) enhances the model's ability to learn complex spatial structures and enhances supervision by applying perturbations and the consistency of nodes and edges. Experimental results demonstrate that the GiTNet outperforms existing state-of-the-art weakly supervised methods across two public datasets. Our code is available at https://github.com/IPMI-NWU/GiTNet.