From Sketch to Reality: Enabling High-Quality, Cross-Category 3D Model Generation From Free-Hand Sketches With...

Area of Science:

• Computer Vision
• Machine Learning
• 3D Graphics

Background:

• Generating high-quality 3D models from sketches is challenging, especially with limited training data.
• Existing methods often focus on single categories and require extensive datasets.
• Free-hand sketches offer an intuitive way to express complex 3D ideas.

Purpose of the Study:

• To develop a novel semi-supervised learning approach for cross-category sketch-to-3D model generation.
• To enable high-quality 3D model creation from limited, imprecise, and abstract sketches.
• To advance the field of 3D content creation and democratize its accessibility.

Main Methods:

• A coarse-to-fine pipeline incorporating semi-supervised learning and a diffusion-based refiner.
• Sketch augmentation techniques and integration of priors like CLIP loss, shape prototypes, and adversarial loss.
• Procedural 3D generation using CAD code for pre-training and curriculum learning for optimization.

Main Results:

• Successful generation of high-quality, cross-category 3D models using as few as 300 sketch-3D pairs.
• Significant improvement over previous single-category approaches.
• Introduction of the KO2D dataset, the largest collection of hand-drawn sketch-3D pairs.

Conclusions:

• The proposed semi-supervised method effectively generates diverse 3D models from limited sketch data.
• This work significantly advances sketch-to-3D conversion, making 3D content creation more accessible.
• The KO2D dataset will foster further research in sketch-based 3D modeling.