InteractOR: Interacting with Images during Surgery in Mixed Reality Through Real-Time Instrument Segmentation

In minimally-invasive surgery, consulting preoperative images such as MRI or CT scans often diverts surgeons' attention from the operative field, leading to interruptions and increasing operative time. To overcome this, we propose InteractOR, a mixed-reality system that overlays preoperative images onto endoscopic video and enables surgeons to interact with these images using their surgical instruments. InteractOR integrates (i) real-time laparoscopic instrument segmentation using TernausNet-16 trained on the CholecInstanceSeg dataset, (ii) a mixed-reality interface that overlays preoperative images onto endoscopic video and (iii) an interaction technique that enables surgeons to control user interfaces using their instruments. An evaluation of the segmentation model is conducted, as well as a usability evaluation of the system. These results demonstrate the feasibility of the approach and its potential for interaction with preoperative images.