Stereo augmented reality in the surgical microscope

We present an augmented reality system that allows surgeons to view features from preoperative radiological images accurately overlaid in stereo in the optical path of a surgical microscope. The purpose of the system is to show the surgeon structures beneath the viewed surface in the correct 3-D position. The technical challenges are registration, tracking, calibration and visualisation. For patient registration, or alignment to preoperative images, we use bone-implanted markers and a dental splint is used for patient tracking. Both microscope and patient are tracked by an optical localiser. Calibration uses an accurately manufactured object with high contrast circular markers which are identified automatically. All ten camera parameters are modelled as a bivariate polynomial function of zoom and focus. The overall system has a theoretical overlay accuracy of better than 1 mm. Implementations of the system have been tested on seven patients. Recent measurements in the operating room conformed to our accuracy predictions. For visualisation the system has been implemented on a graphics workstation to enable high frame rates with a variety of rendering schemes. Several issues of 3-D depth perception remain unsolved, but early results suggest that perception of structures in the correct 3-D position beneath the viewed surface is possible.