Please follow the rules: surgical workflow recognition constrained by linear temporal logic

Surgical workflow recognition aims at automatically recognizing the actions performed during a surgery. Deep learning methods showed their capacity to recognize these actions, but with heavy reliance on training data only, which can sometimes lead to unrealistic predictions. Using procedural knowledge in the form of logic-based rules during model training can help improve temporal consistency and robustness, and also provides a structured way to inspect model outputs.

In this paper, we used differentiable temporal logic (DTL) to define procedural constraints on the action classes of different surgeries, and used them during training to help the network follow these constraints. We also propose 2 new operators taken from Allen's Interval Algebra (AIA) to define more fine-grained and interpretable constraint types and their corresponding evaluation functions. The methods were evaluated using accuracy and segment-based metrics like Edit score and segmental F1-score on 3 different datasets: an in-house dataset of robotic-assisted hysterectomies and 2 other public datasets, namely AutoLaparo and Cholec80. In addition we implemented and compared 3 different types of temporal models: Bi-LSTMs , MS-TCNs, and a transformer-based model (ASFormer).

Both DTL and AIA constraints had a moderate impact on accuracy when observed, but show a trend toward an increase in the segment-based metrics, indicating a higher temporal consistency of the predictions across different datasets and temporal modeling strategies. Statistical testing reveals significant differences in some conditions.

Logic formulas are an effective way of defining procedural constraints on actions, and their use can help during training to obtain temporally consistent predictions, and may support future work on explainability for surgical AI systems. While improvements in frame-wise accuracy remain modest, the results indicate that integrating procedural knowledge can meaningfully influence sequence-level predictions.