OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Graphics

Background:

• Diffusion models excel in 3D scene generation and perception but are typically separated.
• Existing methods often use synthetic data augmentation for perception tasks, limiting integration.

Purpose of the Study:

• To propose OccScene, a unified framework for integrated 3D perception and generation.
• To achieve synergistic improvements in both generation quality and perception accuracy through mutual learning.

Main Methods:

• Developed OccScene, a joint-training diffusion framework guided by semantic occupancy and text prompts.
• Introduced a Mamba-based Dual Alignment module to integrate fine-grained semantics and geometry as perception priors.
• Enabled mutual learning where generation improves perception and vice-versa.

Main Results:

• OccScene generates realistic and consistent 3D scenes from text prompts.
• Demonstrated substantial performance improvements in semantic occupancy prediction.
• Validated effectiveness across diverse indoor and outdoor scenarios.

Conclusions:

• OccScene successfully integrates 3D scene generation and perception in a single framework.
• The mutual learning paradigm offers significant benefits for both tasks.
• Presents a new direction for developing advanced 3D understanding and creation systems.