TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt.

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Graphics

Background:

• Generative models have advanced text-to-3D generation, often using Score Distillation Sampling (SDS).
• SDS faces challenges with multi-condition inputs (text, visual prompts) for customized 3D generation.
• Existing SDS methods exhibit limitations in quality and customization due to their optimization process.

Purpose of the Study:

• To analyze the limitations of Score Distillation Sampling (SDS) in multi-condition text-to-3D generation.
• To propose a novel algorithm, Classifier Score Matching (CSM), to improve the quality and customization of 3D generation.
• To develop an enhanced framework, TV-3DG, integrating visual prompts and semantic-geometry calibration.

Main Methods:

• Decomposition of SDS into difference and classifier-free guidance terms to identify core issues.
• Development of Classifier Score Matching (CSM) by removing the SDS difference term and employing deterministic noise addition.
• Integration of visual prompts via an attention fusion mechanism and sampling guidance, forming Visual Prompt CSM (VPCSM).
• Introduction of a Semantic-Geometry Calibration (SGC) module for enhanced textual information integration.

Main Results:

• CSM effectively overcomes low-quality limitations associated with SDS in customized generation.
• VPCSM successfully integrates visual prompt information for improved 3D generation.
• The Semantic-Geometry Calibration (SGC) module enhances 3D model quality through better textual integration.
• TV-3DG demonstrates stable and high-quality customized 3D generation capabilities.

Conclusions:

• The proposed TV-3DG framework, based on CSM, VPCSM, and SGC, significantly advances customized text-to-3D generation.
• Our approach addresses key limitations of SDS, enabling more robust and high-fidelity 3D content creation.
• TV-3DG offers a stable and effective solution for generating high-quality, customized 3D models from multi-modal inputs.