Automatic preoperative 3d model registration in laparoscopic liver resection

: Augmented Reality (AR) in Laparoscopic Liver Resection requires anatomical landmarks and the silhouette to be found on the laparoscopic image. They are used to register the preoperative 3D model obtained from CT segmentation. The existing AR systems rely on the surgeon to 1) annotate the landmarks and silhouette and 2) provide an initial registration. These non-trivial tasks require surgeon attention which may perturb the procedure. We propose methods to solve both tasks, hence registration, automatically.

: The landmarks are the lower ridge and the falciform ligament. We solve 1) by training a U-Net from a new dataset of 1415 labelled images extracted from 68 procedures. We solve 2) by a novel automatic coarse-to-fine pose estimation method, including visibility-reasoning within an iterative robust process. In addition, we propose to divide the ridge into six anatomical sub-parts, making its annotation and use in registration more accurate.

: Our method detects the silhouette with an error equivalent to an experienced surgeon. It detects the ridge and ligament with higher errors owing to under-detection. Nonetheless, our method successfully initialises the registration with tumour target registration errors of 22.4, 14.8 and 7.2 mm for 3 clinical procedures. In comparison, the errors from manual initialisation are 30.5, 15.1 and 16.3 mm.

: Our results are promising, suggesting that we have found an appropriate methodological approach.