DreamStone: Image as a Stepping Stone for Text-Guided 3D Shape Generation.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• Generating 3D shapes from text descriptions is challenging due to the lack of large-scale paired text-3D datasets.
• Existing methods often struggle with fidelity and consistency between text prompts and generated 3D models.

Purpose of the Study:

• To introduce DreamStone, a novel text-guided 3D shape generation approach.
• To enable 3D shape generation without requiring paired text-3D data by leveraging intermediate image representations.
• To enhance the diversity, fidelity, and stylization capabilities of generated 3D shapes.

Main Methods:

• A two-stage feature-space alignment strategy using a pre-trained single-view reconstruction (SVR) model.
• Mapping CLIP image features to the SVR model's 3D shape space.
• Mapping CLIP text features to the 3D shape space via CLIP-consistency with rendered images.
• Integrating a text-guided 3D shape stylization module.
• Exploiting pre-trained text-to-image diffusion models for enhanced generation.

Main Results:

• DreamStone successfully generates 3D shapes from text descriptions without paired data.
• The approach demonstrates superior generative quality and consistency compared to state-of-the-art methods.
• The stylization module enhances generated shapes with novel structures and textures.
• Integration with diffusion models improves generative diversity and fidelity.

Conclusions:

• DreamStone offers a flexible and scalable solution for text-guided 3D shape generation.
• The method effectively bridges the gap between text and 3D shape modalities.
• The approach shows significant potential for various applications in 3D content creation and AI research.