NeRF-MIR: Toward High-Quality Restoration of Masked Images With Neural Radiance Fields

Neural radiance fields (NeRFs) have demonstrated remarkable performance in novel view synthesis. However, there is much room for restoring 3-D scenes based on NeRF from corrupted images, which are common in natural scene captures and can significantly impact the effectiveness of NeRF. This article introduces NeRF-MIR, a novel neural rendering approach specifically proposed for the restoration of masked images, demonstrating the potential of NeRF in this domain. Recognizing that randomly emitting rays to pixels in NeRF may not effectively learn intricate image textures, we propose a patch-based entropy for ray emitting (PERE) strategy to distribute emitted rays properly. This enables NeRF-MIR to fuse comprehensive information from images of different views. In addition, we introduce a progressively iterative restoration (PIRE) mechanism to restore the masked regions in a self-training process. Furthermore, we design a dynamically weighted loss function that automatically recalibrates the loss weights for masked regions. As existing datasets do not support NeRF-based masked image restoration, we construct three masked datasets to simulate corrupted scenarios. Extensive experiments on real data and constructed datasets demonstrate the superiority of NeRF-MIR over its counterparts in masked image restoration.