RISE-Editing: Rotation-invariant neural point fields with interactive segmentation for fine-grained and efficient...

Area of Science:

• Computer Vision
• Computer Graphics
• Machine Learning

Background:

• Neural Radiance Fields (NeRF) excel at novel view synthesis but lack efficient and fine-grained editing capabilities.
• Existing controllable NeRF methods face limitations in editing efficiency and granularity, restricting creative applications.
• Editing implicit fields is challenging due to potential rendering quality degradation when explicit representations are altered.

Purpose of the Study:

• To present a novel rotation-invariant neural point field representation for enhanced fine-grained and efficient editing of NeRF.
• To enable intuitive and precise manipulation of implicit scene representations.
• To improve the creative editing abilities and potential applications of NeRF.

Main Methods:

• Introduced a novel rotation-invariant neural point field representation by combining implicit NeRF and explicit point-based methods.
• Developed a Rotation-Invariant Neural Inverse Distance Weighting Interpolation (RNIDWI) module for aggregating neural points.
• Disentangled NeRF into scene-agnostic rendering and scene-specific neural point fields for cross-scene compositing.
• Implemented a multi-view ensemble learning strategy for real-time 3D segmentation from 2D prompts.

Main Results:

• Achieved significant improvements in scene rendering quality after fine-grained editing due to local content learning in Cartesian coordinates.
• Enabled efficient 3D neural point field segmentation and manipulation via simple click-based prompts on 2D images.
• Demonstrated enhanced editing capabilities, simplified user editing processes, and photorealistic rendering quality for novel views.
• Outperformed related methods in space-time efficiency and the range of achievable editing functions.

Conclusions:

• The proposed rotation-invariant neural point fields significantly advance fine-grained editing for NeRF.
• The method offers a more intuitive and efficient workflow for users to edit complex 3D scenes.
• This work broadens the practical applications of NeRF by overcoming previous editing limitations.