SEDCLIP: Adapting vision-language model for multi-label surgical error detection

Automated detection of surgical errors can drastically increase safety in robotic-assisted surgery (RAS). Prior work has shown surgical error detection only as a binary task, which is insufficient and restrictive when multiple, co-occurring errors inherently arise in real RAS procedures. We thus present the first work to study multi-label surgical error detection (MLSED) and propose SEDCLIP, a novel vision-language adaptation paradigm that transfers pretrained knowledge from a contrastive language-image pretraining model to MLSED. The SEDCLIP architecture comprises: (1) global-local prompting, which aligns global image features and fine-grained patch features with learnable global and local text prompts, countering the single-label bias of contrastive pre-training; (2) a graph-attention temporal adaptor that models inter-frame dependencies, guided by a temporal-distance adjacency; and (3) random-basis low-rank adaptation, which aggregates fixed low-rank random bases modulated by learnable scaling, to bridge the natural-to-surgical domain gap with efficient fine-tuning. Additionally, to catalyse progress in MLSED, we release two first-of-their-kind, in-vivo datasets with frame-level error annotations: MLE-RAH for robotic-assisted hysterectomy (RAH) and MLE-RARP for robotic-assisted radical prostatectomy (RARP). Across both datasets, SEDCLIP yields overall improvements over competing methods on both multi-label and binary surgical error detection tasks. Code and datasets are available at https://github.com/wzjialang/SEDCLIP.