FLex: joint pose and dynamic radiance fields optimization for stereo endoscopic videos

Reconstruction of endoscopic scenes is crucial for various medical applications, from post-surgery analysis to educational training. However, existing methods are limited by static endoscopes, restricted deformation, or dependence on external tracking devices for camera pose information.

We present flow-optimized local hexplanes (FLex), a novel approach addressing the challenges of a moving stereo endoscope in a highly dynamic environment. FLex implicitly separates the scene into multiple overlapping 4D neural radiance fields (NeRFs) and employs a progressive optimization scheme for joint reconstruction and camera pose estimation from scratch.

Tested on sequences of length up to 5000 frames, which is five times the length handled in the experiments of previous methods, this technique enhances usability substantially. It scales highly detailed reconstruction capabilities to significantly longer surgical videos, all without requiring external tracking information.

Our proposed approach overcomes key limitations of existing methods by enabling accurate reconstruction and camera pose estimation for moving stereo endoscopes in challenging surgical scenes. FLex's advancements enhance the applicability of neural rendering techniques for medical applications, paving the way for improved surgical scene understanding. Code and data will be released on the project page: https://flexendo.github.io/.