MIS-NeRF: neural radiance fields in minimally-invasive surgery

Minimally-invasive surgery (MIS) reduces the trauma compared to open surgery but is challenging for endophytic lesion localisation. Augmented reality (AR) is a promising assistance, which superimposes a preoperative 3D lesion model onto the MIS images. It requires solving the difficult problem of 3D model to MIS image registration. We propose MIS-NeRF, a neural radiance field (NeRF) which provides high-fidelity intraoperative 3D reconstruction, used to bootstrap iterative closest point (ICP) registration.

Existing NeRF methods break down in MIS because of the moving light source and specular highlights. The proposed MIS-NeRF is adapted to these conditions. First, it incorporates the camera centre as an additional input to the radiance function, which allows MIS-NeRF to handle the moving light source. Second, it uses a modified volume rendering which handles specular highlights. Third, it uses a regularised compound loss to enhance surface reconstruction.

MIS-NeRF was tested on three synthetic datasets and retrospectively on four laparoscopic surgeries. It successfully reconstructed high-fidelity liver and uterus surfaces, reducing common artefacts including high-frequency noise and bumps caused by specular highlights. ICP registration achieved faithful alignment between the preoperative and intraoperative 3D models, with an average error of 3.25 mm, outperforming the second-best method by a $15\%$ margin.

MIS-NeRF improves AR-based lesion localisation by facilitating accurate 3D model registration to multiple MIS images.