NaviNeRF++: Towards Interpretable 3D Reconstruction via Unsupervised Disentangled Representation Learning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• 3D reconstruction is crucial for AI's real-world interaction but faces challenges in semantic understanding and prior reliance.
• Existing methods struggle with interpreting the semantics of 3D data and require extensive prior knowledge for control.

Purpose of the Study:

• To propose NaviNeRF++, an interpretable 3D reconstruction framework.
• To address limitations in semantic understanding and prior dependency in 3D reconstruction.
• To achieve fine-grained 3D disentanglement and high-quality reconstruction.

Main Methods:

• Integration of multimodal large language models (MLLMs) and Neural Radiance Fields (NeRF).
• A lightweight 2D perception module for a disentangled latent space, informed by a pre-trained disentangled representation learning (DRL) model.
• A NeRF-based 3D navigation module for semantic factor discovery and high-quality reconstruction.
• An attribute identification module leveraging MLLMs for textual concept identification of semantic factors.

Main Results:

• The framework achieves interpretable 3D reconstruction and fine-grained 3D disentanglement in an unsupervised manner.
• Preserves high-quality and view-consistent 3D reconstruction.
• Demonstrates superior performance compared to existing solutions in empirical evaluations.

Conclusions:

• NaviNeRF++ offers a novel approach to unsupervised, interpretable 3D reconstruction.
• The integration of MLLMs and NeRF advances the field of 3D semantics and disentanglement.
• This framework provides a robust solution for AI's understanding and interaction with 3D environments.