Surgical instrument-tissue characterization via multi-task self-supervised instrument segmentation and motion...

Surgical instrument recognition and tissue motion analysis are two critical tasks for instrument-tissue characterization (ITC) in robot-assisted minimally invasive surgery, with broad applications in surgical scene understanding and 3D reconstruction. While precise instrument segmentation and detailed motion map estimation are the foundation of these tasks, existing methods commonly address these two tasks independently, and little attention has been paid to the fusion of these two tasks for exploring the intrinsic relationships between spatial segmentation and temporal motion. To address this, we propose ITCNet, a multi-task self-supervised instrument segmentation and motion estimation network for instrument-tissue characterization, which simultaneously predicts motion maps and instrument masks from consecutive images. The proposed ITCNet is built upon a shared feature encoder for extracting unified representations and two task-specific decoders for each individual task. To dive into the inherent relationships between spatial segmentation and temporal motion, several fusion constraints are designed to facilitate the multi-task training process, including a motion-based geometric segmentation consistency and a segmentation-based edge-aware motion consistency. Meanwhile, the proposed multi-task fusion scheme enables the adaptation across different surgical scenes with few-shot and zero-shot learning strategies. Extensive experiments have been conducted on three public datasets for comparison with existing methods. Quantitative and qualitative results demonstrate that our proposed method can accurately segment surgical instruments and predict tissue motions across various surgical scenarios. Furthermore, application results on our established surgical robotic system emphasize its potential for real-world surgical robotics.