4DWeaver: Bridging Reconstruction and Generation Via Compact Autoregressive Priors

Large-scale 4D scene generation aims to synthesize dynamic 3D environments and provides a critical intermediate representation for downstream tasks such as autonomous driving simulation, embodied agent training, and scene forecasting. Existing methods typically adopt a two-stage latent diffusion paradigm, which improves computational efficiency by modeling and generating scenes in a compressed latent space. However, this paradigm suffers from a reconstruction-generation trade-off: increasing the latent dimensionality improves reconstruction fidelity, but substantially increases the computational burden of diffusion modeling and makes generative optimization more challenging. This issue becomes particularly pronounced in 4D occupancy generation, where complex spatial layouts and long-range temporal dynamics must be jointly preserved within compact representations. To alleviate this problem, we advance a central principle: low-dimensional latent spaces should not rely solely on unconstrained compression, but should instead be structurally regularized to preserve sufficient 4D spatio-temporal information while maintaining compactness. To instantiate this principle, we propose Compact Autoregressive Latent Prior (CALP), which regularizes low-dimensional latent variables with a history-conditioned autoregressive prior, achieving compact, long-range temporally coherent, and spatially structured latent representations through a more reasonable latent capacity allocation and a latent-space organization that is better suited for diffusion-based 4D generation. Built upon CALP, we further introduce 4DWeaver, a compact 4D scene generation framework that enables high-quality spatio-temporal occupancy synthesis in a low-dimensional latent space. Extensive experiments on multiple large-scale 4D occupancy benchmarks demonstrate that 4DWeaver achieves superior reconstruction and generation performance while substantially reducing memory consumption and computational cost.