Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generation

Area of Science:

• Computer Vision
• Machine Learning
• 3D Graphics

Background:

• Advances in 2D text-to-image diffusion models are rapid due to abundant 2D data.
• 3D diffusion model development lags due to limited high-quality 3D data, impacting performance.
• Existing 3D generative models face challenges with data scarcity and computational complexity.

Purpose of the Study:

• To address the scarcity of 3D data for diffusion models.
• To adapt powerful pre-trained 2D diffusion models for 3D object generation.
• To develop a novel representation and dataset for efficient 3D diffusion model training.

Main Methods:

• Introduced Gaussian Atlas, a novel representation using dense 2D grids for 3D Gaussian generation.
• Fine-tuned pre-trained 2D diffusion models on a 2D manifold derived from 3D structures.
• Compiled GaussianVerse, a large-scale dataset of 205K 3D Gaussian fittings for model training.

Main Results:

• Demonstrated successful transfer learning from 2D to 3D diffusion models.
• Generated high-quality 3D Gaussians using adapted text-to-image models.
• Showcased the effectiveness of Gaussian Atlas and GaussianVerse for 3D content generation.

Conclusions:

• Pre-trained 2D diffusion models can be effectively repurposed for 3D object generation.
• The proposed method bridges the performance gap between 2D and 3D diffusion models.
• Gaussian Atlas and GaussianVerse offer a viable solution for large-scale 3D generative modeling.