Augmented reality endoscopic system (ARES): preliminary results

During endoscopic surgery, it is difficult to ascertain the anatomical landmarks once the anatomy is fiddled with or if the operating area is filled with blood. An augmented reality system will enhance the endoscopic view and further enable surgeons to view hidden critical structures or the results of preoperative planning.

The skull and endoscope are fixed with optical markers that are used as dynamic reference bases for tracking. A small optical tracking device, the easyTrack 200, which is connected to a computer, calculates the positions of the markers. The endoscope is calibrated and registered for augmenting its video with a 3D model. Images of a black and white checkerboard pattern, with 2.5 mm sized squares, are used for calibration with a Matlab based calibration toolbox. Standard modalities of overlay have been developed, including a CT viewer displaying it as an overlay in the endoscopic video stream, and a 3D viewer to render 3D models of preoperatively segmented structures. The accuracy of the augmented reality system was assessed on a plastic skull.

The accuracy is calculated by looking at the difference in pixels of several contours in both a real and an overlay image, obtaining a mean of 3-4 pixels that correspond to sub-millimeter accuracy (pixel to mm ratio calculated previously). Mean error was consistently 1-2 [+/- 0.3] mm.

A novel augmented reality system for endoscopic surgery is presented. Highlighting hidden structures or CT overlays in the endoscope will give more information in difficult situations and enhance the operation quality.