Real-time tracking and inpainting network with joint learning iterative modules for AR-based DALK surgical navigation

Area of Science:

• Ophthalmology
• Computer Vision
• Medical Imaging

Background:

• Deep anterior lamellar keratoplasty (DALK) requires precise suturing for successful outcomes.
• Augmented Reality (AR) navigation systems offer potential to improve surgical accuracy.
• Clear visualization of corneal regions is crucial for planning stitch placement in DALK.

Purpose of the Study:

• To develop a joint-learning and iterative network for AR-based suturing navigation.
• To enhance inpainting performance under severe occlusion during the suturing process.
• To provide surgeons with clear, unoccluded views of corneal regions for better surgical planning.

Main Methods:

• A novel joint-learning and iterative network was designed for AR-based suturing navigation.
• The network utilizes feature reuse, iterative modules, and mask propagation for computational efficiency.
• A synthetic dataset with occluded/unoccluded image pairs and annotations was created for end-to-end training.
• A pipeline using grid propagation and inpainted optical flow was developed for stable frame generation.

Main Results:

• The Iter-S model achieved a mean endpoint error (mEPE) of 1.69, PSNR of 36.86, and SSIM of 0.976.
• Inpainting inference time was as low as 16.26ms, demonstrating computational efficiency.
• The developed AR navigation system achieved an average frame rate of 28 FPS.
• The framework showed a better trade-off between performance and computational efficiency compared to existing inpainting networks.

Conclusions:

• Iterative modules progressively refine outputs, balancing visual performance and real-time efficiency.
• The AR navigation framework provides stable, accurate tracking with real-time, well-inpainted results under severe occlusion.
• This system demonstrates significant benefits for guiding corneal surgical stitching operations.