UrbanGen: Urban Generation with Compositional and Controllable Neural Fields.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Generative radiance fields have advanced rapidly but struggle with complex urban scene generation.
• Existing methods primarily focus on object-centric applications, lacking capabilities for intricate city environments.

Purpose of the Study:

• To introduce UrbanGen, a novel solution for generating high-fidelity urban radiance fields.
• To achieve photorealistic rendering, accurate geometry, and high controllability in complex urban scenes.
• To enable diverse city style generation and editing.

Main Methods:

• Leveraging a coarse 3D panoptic prior (semantic voxel grids for stuff, bounding boxes for objects) to condition a compositional generative radiance field.
• Employing semantic and geometry losses combined with adversarial training for adherence to input conditions.
• Developing a unified dataset from KITTI-360, nuScenes, and Waymo, incorporating panoptic priors for training a city style-aware model.

Main Results:

• UrbanGen demonstrates superior performance over state-of-the-art baselines in image fidelity and geometry accuracy for urban scene generation.
• The method successfully generates urban radiance fields with photorealistic rendering and accurate geometric representations.
• Achieved joint rendering of RGB images, semantic maps, and depth maps.

Conclusions:

• UrbanGen effectively addresses the challenge of generating complex urban radiance fields.
• The proposed panoptic prior enables disentanglement and versatile control over scene elements and styles.
• UrbanGen introduces new controllability features, including large camera movements, stuff editing, and city style manipulation, advancing the field of generative scene synthesis.