Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis.

Area of Science:

• Computer Vision
• Deep Learning
• Generative Models

Background:

• Deep learning success relies on large datasets and pretrained models, especially in data-scarce settings.
• Traditional pretraining uses image classification and self-supervised learning.
• Text-to-image diffusion models offer a new class of foundational models with visual understanding.

Purpose of the Study:

• To present Marigold, a method for adapting latent diffusion models for dense image analysis.
• To enable effective transfer learning from large-scale generative models to specific vision tasks.

Main Methods:

• Developed Marigold, a family of conditional generative models.
• Introduced a fine-tuning protocol to extract knowledge from pretrained latent diffusion models (e.g., Stable Diffusion).
• Adapted models for tasks like monocular depth estimation, surface normals prediction, and intrinsic decomposition.

Main Results:

• Marigold requires minimal architectural modification of pretrained models.
• Training is efficient, using small synthetic datasets on a single GPU within days.
• Achieved state-of-the-art zero-shot generalization performance on dense image analysis tasks.

Conclusions:

• Latent diffusion models can be effectively repurposed for dense computer vision tasks.
• Marigold provides a computationally efficient and high-performing method for knowledge transfer.
• This work demonstrates the potential of generative models as powerful backbones for various vision applications.