FLex: joint pose and dynamic radiance fields optimization for stereo endoscopic videos.

Area of Science:

• Medical imaging
• Computer vision
• Surgical technology

Background:

• Endoscopic scene reconstruction is vital for medical training and analysis.
• Current methods struggle with dynamic environments, limited deformation, and reliance on external tracking.

Purpose of the Study:

• To develop a novel approach for reconstructing moving stereo endoscope scenes in dynamic environments.
• To overcome limitations of existing methods by enabling accurate reconstruction and camera pose estimation without external tracking.

Main Methods:

• Introduced flow-optimized local hexplanes (FLex).
• Implicitly separates scenes into overlapping 4D neural radiance fields (NeRFs).
• Employs progressive optimization for joint reconstruction and camera pose estimation from scratch.

Main Results:

• Tested on sequences up to 5000 frames, five times longer than previous methods.
• Achieved highly detailed reconstruction for significantly longer surgical videos.
• Eliminated the need for external tracking information.

Conclusions:

• FLex enables accurate reconstruction and camera pose estimation for moving stereo endoscopes in challenging surgical scenes.
• Advances applicability of neural rendering for improved surgical scene understanding.
• Code and data will be released.