NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos

Area of Science:

• Computer Vision
• Medical Imaging
• Robotics

Background:

• Operating room (OR) cameras provide synchronized multi-view data for surgical scene analysis.
• Neural Radiance Fields (NeRFs) offer memory-efficient 3D scene representations.
• Existing NeRF methods struggle with sparse-view data typical in OR settings.

Purpose of the Study:

• To propose NeRF-OR, a novel method for 3D reconstruction of dynamic surgical scenes.
• To address the challenges of sparse-view data acquisition in real-world OR environments.
• To improve downstream tasks like object detection and pose estimation in surgery.

Main Methods:

• NeRF-OR combines time-of-flight sensor depth and estimated depth from color images.
• Depth estimations are used to mitigate missing values from reflective surfaces and object boundaries.
• Supervision is applied using surface normals derived from estimated depths for scale invariance.

Main Results:

• NeRF-OR achieves geometrically accurate representations in static surgical scenes, surpassing state-of-the-art methods.
• The method demonstrates 30x faster training and captures fine scene details.
• NeRF-OR synthesizes intermediate video views with an average PSNR of 24.86 dB and shows promise in non-OR sparse-view settings.

Conclusions:

• NeRF-OR facilitates novel view synthesis from limited, widely-angled camera views common in ORs.
• The approach offers a viable solution for 3D reconstruction in challenging surgical environments.
• Code is publicly available for further research and application.