IS-Diff: Improving Diffusion-Based Inpainting With Better Initial Seed

Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs have produced realistic results with improved contextual coherence between the inpainted regions and the visible image content. However, the random initial noise/state adopted in the vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low semantic coherence and inconsistent appearance with the unmasked regions. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff constructs a primary seed initialization by sampling from the unmasked regions to approximate the masked-region data distribution, and then combines it with random noise to form a compatible initial noise/state, thereby setting a promising direction for the inpainting. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.