TPDPM: Text promptable diffusion probabilistic model for referring surgical instrument segmentation

Referring surgical instrument segmentation (RSIS) plays a vital role in advancing context-aware computer-assisted intervention systems. Unlike conventional vision-based surgical instrument segmentation, which typically segments all instruments simultaneously, RSIS focuses on segmenting the target instrument given a referring textual expression, thereby enabling target-specific segmentation and enhancing interactive capability to meet practical demands of robot-assisted surgery. Existing RSIS methods are developed primarily based on discriminative per-pixel classification learning. In contrast, diffusion probabilistic models, as a class of generative models, have demonstrated promising results across various segmentation tasks. To this end, we propose a novel Text Promptable Diffusion Probabilistic Model, referred to as TPDPM, which for the first time formulates the solution to RSIS as a mask generation task conditioned on both surgical images and referring textual expressions where the segmentation mask of a target instrument is generated through a reverse diffusion process starting from a noisy mask. Benefiting from the inherent stochastic sampling process of diffusion probabilistic model, TPDPM can effectively handle ambiguous instrument boundaries by ensembling multiple network predictions for accurate RSIS and uncertainty assessment. The proposed method is thoroughly validated on two publicly available porcine video datasets, i.e., the EndoVis-RS17 dataset and the EndoVis-RS18 dataset, and one in-house collected patient video dataset. Results obtained from these comprehensive experiments demonstrate that our method achieves not only superior RSIS performance when compared with other competing state-of-the-art methods but also competitive generalization capability across different datasets and domains. The source code and the dataset of this study will be released at: https://github.com/guascy666/TPDPM.