Acquiring Aligned Endoscopic and Depth Image Pairs Using Structured-Light Projection, Neural Surfaces and an...

Area of Science:

• Medical Imaging
• Computer Vision
• Robotics

Background:

• Minimally invasive endoscopic procedures increasingly require 3D data for enhanced diagnostic and surgical accuracy.
• Existing 3D endoscopic systems, particularly those using structured-light (SL) projection, face challenges like pattern interference and inaccurate pose estimation.
• Removing projected patterns is essential for obtaining clean texture images crucial for medical diagnosis.

Purpose of the Study:

• To develop an advanced 3D endoscopy approach addressing limitations of current SL-based systems.
• To integrate neural signed distance fields (neural SDF) with SL projection for robust 3D reconstruction.
• To enable accurate 3D shape measurement and pattern removal in endoscopic imaging.

Main Methods:

• Camera pose estimation using electromagnetic sensors for accurate initializations.
• 3D shape measurement through SL projection integrated with a neural implicit surface representation.
• Multi-frame shape integration and pattern removal from endoscopic images using a pix2pix model.
• Optimization of camera pose, projector pose, and surface geometry within a simulated neural surface model.

Main Results:

• Achieved accurate initial camera poses via electromagnetic tracking, enhancing optimization robustness.
• Successfully integrated multi-frame shape data using neural SDF for detailed 3D reconstruction.
• Demonstrated effective removal of SL projection patterns, yielding clean texture images.
• Generated paired sequences of endoscopic images and corresponding depth maps.

Conclusions:

• The proposed neural SDF-based 3D endoscopy method overcomes key challenges in SL projection, including pattern interference and pose estimation failures.
• This approach enables high-quality 3D reconstruction and clean texture image generation, crucial for advancing medical diagnosis and surgical guidance.
• The integration of electromagnetic tracking and neural implicit representations offers a robust solution for practical 3D endoscopy.