AnyDoor: Zero-Shot Image Customization With Region-to-Region Reference

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Object insertion and manipulation in images are complex tasks.
• Existing methods often require object-specific fine-tuning or struggle with generalization.
• Controllable image generation with precise object placement remains a significant challenge.

Purpose of the Study:

• To introduce AnyDoor, a diffusion-based image generation model for zero-shot object teleportation.
• To enable precise control over object placement, shape, and integration into new scenes.
• To develop a unified framework for object insertion, removal, and image variation.

Main Methods:

• Leveraging DINOv2 for discriminative object identity feature extraction.
• Complementing identity features with detail features for appearance consistency and local variations.
• Utilizing video datasets to enhance model generalizability and robustness.
• Extending the framework for region-to-region image referencing, unifying multiple generation tasks.

Main Results:

• AnyDoor demonstrates effective zero-shot generalization across diverse object-scene combinations.
• The model successfully teleports objects to specified locations with desired shapes.
• A unified model handles object insertion, removal, and image variation without additional parameters.
• Incorporation of masks, pose skeletons, and depth maps allows for more controllable generation.

Conclusions:

• AnyDoor provides a powerful and versatile solution for controllable object manipulation in images.
• The proposed method significantly advances the state-of-the-art in zero-shot image generation and object insertion.
• The unified framework offers a flexible approach for various image editing and generation tasks.