Collaborative Multi-Modal Coding for High-Quality 3D Generation.

Area of Science:

• Computer Vision
• Computer Graphics
• Artificial Intelligence

Background:

• 3D content generation often relies on single-modality data, neglecting complementary information from other sources.
• Existing 3D generative models have limitations in harnessing multi-modal data effectively.

Purpose of the Study:

• To introduce TriMM, the first feed-forward 3D-native generative model capable of learning from multiple modalities.
• To develop a method that holistically leverages multi-modal data for enhanced 3D asset modeling.

Main Methods:

• Collaborative multi-modal coding to integrate modality-specific features while preserving unique strengths.
• Auxiliary 2D and 3D supervision to improve the robustness and performance of multi-modal coding.
• A triplane latent diffusion model for generating high-quality 3D assets with enhanced texture and geometry.

Main Results:

• TriMM achieves competitive performance compared to models trained on large datasets, despite using limited training data.
• The model effectively leverages multi-modality to enhance both texture and geometric detail in generated 3D assets.
• Demonstrated feasibility of incorporating diverse multi-modal datasets into 3D generation.

Conclusions:

• TriMM offers a powerful new approach for 3D content generation by effectively utilizing multi-modal data.
• The model's ability to achieve high-quality results with less data has significant implications for efficient 3D asset creation.
• Future work can explore broader applications and integration of various multi-modal datasets.