Hyper-Spherical Optimal Transport for Semantic Alignment in Text-to-3D End-to-End Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• 3D Graphics

Background:

• CLIP-guided 3D generation shows promise but struggles with text-shape consistency.
• A key challenge is the semantic gap between text and image embeddings.

Purpose of the Study:

• To introduce HOTS3D, a novel method for text-to-3D generation.
• To effectively bridge the semantic gap between text and image features.

Main Methods:

• Aligning text and image features using spherical optimal transport (SOT).
• Developing a high-dimensional SOT solution based on Villani's theorem.
• Implementing SOT with input convex neural networks (ICNNs).
• Utilizing a diffusion-based generator for 3D shape decoding.

Main Results:

• HOTS3D demonstrates superior performance in text-to-3D generation.
• Achieved enhanced fidelity and consistency between generated 3D shapes and text semantics.
• Outperformed state-of-the-art methods in quantitative and qualitative evaluations.

Conclusions:

• HOTS3D effectively bridges the text-image embedding gap for improved 3D generation.
• The proposed SOT approach offers a robust solution for high-dimensional feature alignment.
• This work advances the field of text-to-3D generation with greater semantic accuracy.