Enhancing endoscopic scene reconstruction with color-aware inverse rendering through neural SDF and radiance fields

Area of Science:

• Medical Imaging
• Computer Graphics
• Surgical Simulation

Background:

• Virtual surgical training requires realistic 3D organ models.
• Current methods struggle with accurate color and geometric reconstruction from medical images.
• Endoscopic imaging lacks detailed appearance information.

Purpose of the Study:

• To develop a novel network, Endoscope-NeSRF, for simultaneous geometric and appearance reconstruction of organs from endoscopic images.
• To improve the realism of virtual surgical training environments.
• To overcome limitations of traditional reconstruction methods.

Main Methods:

• Utilized neural radiance fields and Signed Distance Function (SDF) for organ model reconstruction.
• Incorporated prior knowledge of light-object interaction and a dilated mask for edge refinement.
• Developed a highlight adaptive optimization strategy to remove specular reflections.
• Employed inverse rendering and Bidirectional Reflectance Distribution Function (BRDF) rendering for real-time visualization.

Main Results:

• Endoscope-NeSRF achieved comparable appearance reconstruction to Instant-NGP.
• Demonstrated superior geometric reconstruction accuracy compared to state-of-the-art methods.
• Generated detailed geometric models with realistic appearance from multi-view endoscopic images.
• Successfully removed highlight artifacts, preventing white-out effects.

Conclusions:

• Endoscope-NeSRF provides a significant advancement in creating realistic 3D organ models for virtual surgical simulation.
• The method enhances the visual fidelity crucial for effective medical training.
• Offers a more accurate and visually rich alternative to traditional reconstruction techniques.