Stream-based active learning for surgical AI

Continuous deployment in surgical AI promises adaptive models that evolve with clinical practice, yet its success critically depends on efficient annotation, balancing limited expert time with the need to identify and label only the most informative data to ensure sustained model improvement. We propose a Stream-based Active Learning pipeline where annotation and retraining extend beyond initial deployment, allowing models to adapt and mitigate distribution shifts. Collecting expert annotations during surgery presents both challenges and opportunities: selecting non-redundant, informative data improves generalization, while immediate integration of expert feedback can refine model predictions in real time.

To enable online selection of informative training data during surgery, we build upon VeSSAL, a Stream-based Active Learning method motivated by model gradients, utilizing both model uncertainty and data diversity in its acquisition function. We improve the sample selection of Active Learning by including an Adversarial Autoencoder (AAE) loss component during training, normalizing the latent features.

Our enhanced version of VeSSAL achieves a stable labeling ratio with 12% standard deviation, while improving the macro F1 score by up to 7.7% after 15 surgeries over equidistant sampling and thresholding strategies of uncertainty and variance. We evaluate on a private dataset and Cholec80.

We demonstrate that intraoperative, Stream-based Active Learning is feasible and label-efficient for continuous deployment. Gradient-based Active Learning combined with an AAE for feature normalization improves model generalization and sample selection, supporting a practical path toward continuously learning for surgical AI systems ( Code ).