Hybrid Rendering Architecture for Realtime and Photorealistic Simulation of Robot-Assisted Surgery

In this paper we present a method for combining realtime and non-realtime (photorealistic) rendering with open source software. Realtime rendering provides sufficient realism and is a good choice for most simulation and regression testing purposes in robot-assisted surgery. However, for proper end-to-end testing of the system, some computer vision algorithms require high fidelity images that capture more minute details of the real scene. One of the central practical obstacles to combining both worlds in a uniform way is creating models that are suitable for both kinds of rendering paradigms. We build a modeling pipeline using open source tools that builds on established, open standards for data exchange. The result is demonstrated through a unified model of the medical OpenHELP phantom used in the Gazebo robotics simulator, which can at the same time be rendered with more visual fidelity in the Cycles raytracer.