A-MFST: adaptive multi-flow sparse tracker for real-time tissue tracking under occlusion

Tissue tracking is critical for downstream tasks in robot-assisted surgery. The Sparse Efficient Neural Depth and Deformation (SENDD) model has previously demonstrated accurate and real-time sparse point tracking, but struggled with occlusion handling. This work extends SENDD to enhance occlusion detection and tracking consistency while maintaining real-time performance.

We use the Segment Anything Model2 (SAM2) [1] to detect and mask occlusions by surgical tools, and we develop and integrate into SENDD an Adaptive Multi-Flow Sparse Tracker (A-MFST) with forward-backward consistency metrics, to enhance occlusion and uncertainty estimation. A-MFST is an unsupervised variant of the Multi-Flow dense Tracker (MFT) [2].

We evaluate our approach on the STIR dataset [3] and demonstrate a significant improvement in tracking accuracy under occlusion, reducing average tracking errors by 12% in Mean Endpoint Error (MEE) and showing a 6% improvement in ${\delta }_{\text{avg}}^x$ , the averaged accuracy over thresholds of [4, 8, 16, 32, 64] pixels [4]. The incorporation of forward-backward consistency further improves the selection of optimal tracking paths, reducing drift and enhancing robustness. Notably, these improvements were achieved without compromising the model's real-time capabilities.

Using A-MFST and SAM2, we enhance SENDD's ability to track tissue in real-time, under instrument and tissue occlusions. Our approach improves tracking accuracy and reliability by integrating SAM2 for robust occlusion handling and employing forward-backward consistency for optimal frame selection. Experimental results on the STIR dataset demonstrate that A-MFST reduces tracking errors while preserving real-time performance, making it well suited for surgical applications. Future work will focus on further refining adaptive mechanisms to enhance robustness and computational efficiency.