SWAG: long-term surgical workflow prediction with generative-based anticipation

Area of Science:

• Computer Vision
• Medical Informatics
• Surgical Robotics

Background:

• Current surgical phase recognition lacks foresight and intraoperative guidance.
• Existing anticipation methods predict short-term, single events, failing to capture complex surgical workflows.

Purpose of the Study:

• To introduce SWAG (surgical workflow anticipative generation), a framework combining phase recognition and anticipation.
• To address limitations in predicting long-term, sequential surgical events.

Main Methods:

• Investigated single-pass (SP) and autoregressive (AR) decoding for generating future surgical phase sequences.
• Proposed a novel embedding using class transition probabilities for enhanced phase anticipation.
• Developed a generative framework (R2C) incorporating remaining time regression to classification.

Main Results:

• The SP* model achieved 32.1% and 41.3% F1 scores over 20 and 30 minutes on Cholec80 and AutoLaparo21 datasets, respectively.
• The approach demonstrated competitive performance in phase remaining time regression, with weighted mean absolute errors of 0.32 and 0.48 minutes for 2- and 3-minute horizons.

Conclusions:

• SWAG offers versatility in generative decoding, classification, and regression tasks.
• The framework establishes temporal continuity between surgical workflow recognition and anticipation.
• SWAG advances intraoperative surgical workflow generation for enhanced anticipation.