Harnessing Diffusion Models for Image Manipulation With Partial Sketches.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Controllable image editing is crucial, but current methods lack fine-grained local control.
• Generative adversarial network (GAN)-based sketch editing has limited generalization and fidelity.
• Diffusion models struggle with localized editing due to sparse stroke injection and preserving non-edited regions.

Purpose of the Study:

• To introduce DiffStroke, a mask-free framework for localized image manipulation using partial sketches.
• To enable precise local shape control and maintain appearance consistency.
• To protect non-edited regions automatically without manual input.

Main Methods:

• Developed trainable Image-Stroke Fusion (ISF) blocks for feature-level fusion of images and strokes.
• Introduced a self-supervised mask estimator leveraging Tweedie's formula for latent image estimation and blending.
• Trained the mask estimator by minimizing the error between blended and target latents.

Main Results:

• DiffStroke outperforms state-of-the-art methods in stroke-based image editing tasks.
• Demonstrated effectiveness on both natural and facial images for simple and complex edits.
• Showcased the ability to combine DiffStroke with text prompts for diverse results.

Conclusions:

• DiffStroke offers a robust solution for mask-free, localized image structure editing.
• The framework achieves precise control and preserves image integrity effectively.
• DiffStroke advances the field of controllable image manipulation with partial sketches.