A Bayesian approach to temporal surgical segmentation model fusion

Robotic-assisted surgery (RAS) generates vast amounts of video and robotic data, presenting opportunities for machine learning. Video-based models, in particular, that can temporally segment frames by ontological categories such as procedure type, phase, steps, actions, etc., are needed. Training separate models for each category neglects statistical dependencies between categories and can yield incompatible predictions. Training large multi-category models may help, but increases complexity while reducing model modularity and interpretability.

We present a model fusion alternative: an effectively zero-free-parameter Bayesian model fusion technique. Incorporating the empirical conditional dependencies across categories and time, we combine predictions from multiple segmentation models into one joint Bayesian inference. The result is a Bayes' optimal distribution over all categories evolving over time with accumulated evidence.

On a large test set of hundreds of surgical cases, of nearly eight million frames of annotated data, we found that fused predictions from the joint Bayesian model provide clear benefits over the individual models, correcting inconsistent and inaccurate predictions, and even forming accurate beliefs when evidence was absent.

The model we present is a lightweight, principled alternative to machine learning-based model fusion. A sufficiently complex model could be trained to produce the same results, but would effectively trade explainable predictions with minimal overheard for computational complexity and transparency. We end by discussing how the same approach can be used to encompass larger more sophisticated models within the same conceptual framework.