DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Graphics

Background:

• Natural language to 3D scene generation is promising but lacks automation, consistency, and control.
• Existing methods face challenges in creating complex and editable 3D environments.

Purpose of the Study:

• To introduce DreamScene, an end-to-end framework for high-quality, editable 3D scene generation from text or dialogue.
• To address limitations in automation, 3D consistency, and fine-grained control in current 3D content creation methods.

Main Methods:

• Utilizes a GPT-4 agent for scene planning, inferring semantics and spatial constraints to build a hybrid graph.
• Employs a graph-based placement algorithm for structured, collision-free layouts and Formation Pattern Sampling (FPS) for object geometry generation.
• Incorporates progressive camera sampling for global consistency and supports fine-grained editing of objects and dynamics.

Main Results:

• DreamScene demonstrates superior quality, consistency, and flexibility compared to existing methods.
• The framework enables fast and realistic synthesis of 3D scenes with editable elements.
• Achieves practical, open-domain 3D content creation from natural language inputs.

Conclusions:

• DreamScene provides a robust solution for automated and controllable 3D scene generation.
• The framework advances the state-of-the-art in text-to-3D content creation.
• Offers a practical tool for gaming, film, and design industries.