Tracking better, tracking longer: automatic keyframe selection in model-based laparoscopic augmented reality

We present a novel automatic system for markerless real-time augmented reality. Our system uses a dynamic keyframe database, which is required to track previously unseen or appearance-changing anatomical structures. Our main objective is to track the organ more accurately and over a longer time frame through the surgery.

Our system works with an offline stage which constructs the initial keyframe database and an online stage which dynamically updates the database with new keyframes automatically selected from the video stream. We propose five keyframe selection criteria ensuring tracking stability and a database management scheme ensuring real-time performance.

Experimental results show that our automatic keyframe selection system based on a dynamic keyframe database outperforms the baseline system with a static keyframe database. An increase in number of tracked frames without requiring surgeon input is observed with an average improvement margin over the baseline of 11.9%. The frame rate is kept at the same values as the baseline, close to 50 FPS, and rendering remains smooth.

Our software-based tracking system copes with new viewpoints and appearance changes during surgery. It improves surgical organ tracking performance. Its criterion-based architecture allows a high degree of flexibility in the implementation, hence compatibility with various use cases.