Urban Architect: Steerable 3D Driving Scene Generation With Layout Prior

Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, scaling this methodology to large-scale driving scenes is underexplored. Driving scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization. In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Specifically, we represent the scene via 2D Gaussian splatting and develop two innovative optimization strategies: (1) Layout-Guided Variational Score Distillation, which addresses model optimization inadequacies by conditioning the score distillation sampling process with geometric and semantic constraints of 3D layouts; (2) Semantic-Depth Score Distillation Sampling, which ensures alignment of semantics, geometry, and appearance by matching the rendered semantic and depth maps with 3D layouts. Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale driving scenes that cover over 1000 m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation.