Cross Domain Self-Prompting SAM2 for Intraoperative OCT Video Segmentation

Retinal procedures require precise instrument manipulation to avoid iatrogenic injuries. Intraoperative Optical Coherence Tomography (iOCT) delivers realtime cross-sectional views of tissue-instrument interactions, enabling safer surgical decision-making. Accurate iOCT segmentation would allow the automated estimation of the distance between surgical instruments and retinal surfaces, and the development of safety alerts for both manual and robotic vitreoretinal procedures. Although existing deep learning models have advanced iOCT segmentation, they operate frame-wise without temporal modeling and are trained on large annotated iOCT datasets acquired under simulated conditions. To overcome these limitations, we propose a cross-domain self-prompting SAM2 framework for real-time iOCT video segmentation (50 FPS). Our method introduces a cross-domain pseudo mask generator that transfers knowledge from large-scale clinical OCT datasets to the annotation-scarce iOCT domain and enables fully automatic video inference without manual prompts (e.g., clicks or boxes). We comprehensively compare our method against state-of-the-art approaches. On two iOCT datasets collected from real vitreoretinal surgeries, our model achieved mean Dice scores of 91.21% and 89.16%, surpassing the second-best method by 4.47% and 0.48%, respectively. We further estimated the toolto- retina distance from segmentation, achieving a mean error of only 17μm, below average human hand tremor (0.182 mm), demonstrating its potential to enhance intraoperative safety. To our knowledge, this is the first SAM2- based framework for fully automatic segmentation of real vitreoretinal iOCT videos without manual prompting. Code is available here.