NeRF-based multi-frame 3D integration for 3D endoscopy using active stereo.

Area of Science:

• Medical Imaging
• Computer Vision
• Robotics

Background:

• Accurate 3D measurement in endoscopy is crucial for cancer diagnosis, computer-assisted interventions, and training deep neural networks.
• Active-stereo systems with micro-pattern projectors offer a promising approach for endoscopic 3D measurement.
• Existing multi-frame optimization methods rely on accurate initial 3D triangle meshes, limiting their practicality in real-world endoscopic systems.

Purpose of the Study:

• To develop a novel 3D scene representation and optimization method for endoscopic active-stereo systems.
• To overcome the dependency on accurate initial 3D triangle meshes in existing methods.
• To improve the accuracy and robustness of 3D endoscopic measurements.

Main Methods:

• Adaptation of Neural Radiance Fields (NeRF) for 3D scene representation within an active-stereo endoscopic framework.
• Simultaneous optimization of the 3D scene and camera/projector poses using multi-frame data.
• Integration of differential rendering for pose and scene optimization without requiring an initial 3D mesh.

Main Results:

• Successful 3D reconstruction using both synthetic and real endoscopic images.
• Demonstrated simultaneous optimization of 3D scene and camera/projector poses.
• Eliminated the need for an accurate initial 3D triangle mesh, enhancing practical applicability.

Conclusions:

• The proposed NeRF-based method enables accurate and robust 3D endoscopic measurement without relying on initial shape information.
• This approach significantly enhances the practicality of active-stereo systems for clinical applications like cancer diagnosis and AI training data generation.
• One-shot endoscopic depth information measurement is a viable solution for various medical applications.