Dynamic Scene Understanding Through Object-Centric Voxelization and Neural Rendering.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Generative Models

Background:

• Unsupervised learning of object-centric representations from videos is difficult.
• Existing methods often focus on 2D image decomposition, limiting 3D scene understanding.

Purpose of the Study:

• To develop a 3D generative model for dynamic scenes that facilitates object-centric learning.
• To enable novel view synthesis and unsupervised decomposition of complex dynamic scenes.

Main Methods:

• Introduced DynaVol-S, a 3D generative model using object-centric voxelization within a differentiable volume rendering framework.
• Integrated 2D semantic features to create 3D semantic grids for disentangled voxel representations.
• Optimized using an inverse rendering pipeline with a compositional Neural Radiance Fields (NeRF).

Main Results:

• DynaVol-S significantly outperforms existing models in novel view synthesis and unsupervised decomposition for dynamic scenes.
• The model effectively handles complex object interactions in real-world scenarios.
• Achieved superior performance by jointly considering geometric structures and semantic features.

Conclusions:

• DynaVol-S enables robust object-centric learning in dynamic scenes, surpassing 2D methods.
• The learned 3D voxel features allow for advanced capabilities like novel scene generation and object manipulation.
• This approach advances the understanding and generation of dynamic 3D environments.