EasyVis: a real-time 3D visualization software system for laparoscopic surgery box trainer

EasyVis is an emerging immersive 3D laparoscopic visualization system for improving the efficiency of laparoscopic surgery. It integrates multiple micro-cameras and light sources with the surgical ports to provide intra-abdominal stereo vision of surgery at a desired viewpoint. In this work, we develop a visualization algorithm using the EasyVis micro-camera array assembly in a laparoscopic surgery training box environment with simplified training tasks to validate the feasibility of this novel technology. Since most laparoscopic surgical tools are rigid-body objects, their 3D shape may be acquired offline. We developed 2D object detection and track algorithms to acquire the 2D pose of each object and a 3D fusion algorithm to estimate and track the 3D pose of each object using estimated 2D poses. Then, together with the acquired 3D model of each object, we are able to render each object at a desired view using the 3D surface model (acquired offline) and images acquired from individual micro-cameras. In addition to the foreground rigid objects, the background 3D model is acquired using structured lights and structure from motion. The background is assumed to be slowly varying compared to the rapid motion of the foreground objects. As such, the background 3D model needs to be updated only occasionally. Our rendering algorithm is capable of integrating the foreground and background 3D models to facilitate image-based rendering from a desirable viewing angle. We performed experiments to validate the accuracy and quality of the rendered images.