Stylized Sketch Synthesis Using Diffusion Priors and Structural Guidance

Translating an image into a reference style is a widely studied problem. Sketches, as a family of highly abstract styles, are usually difficult to synthesize under general image stylization frameworks. A major challenge is that the reference style image could be semantically irrelevant to the content image. Based on recent advances in image stylization using diffusion models, we propose a novel stylized sketch synthesis algorithm that directly manipulates the latent code of the variational autoencoder using the priors of a pretrained diffusion model. In particular, we construct a unified attention distillation loss to align the self-attention layer features with content and style images. To maintain the semantic structures in the content image, we introduce a structural guidance for additional regularization. As for the guidance map, we adopt a dynamic vector sketch that is jointly optimized with the latent code. We also explore possible alternatives: using a fixed Canny edge map or using multiple guidance maps. We evaluated our method on a Diverse-Sketch-Stylization (DSS) dataset that includes various reference sketch styles and also conducted experiments on 4SKST, FS2K, and Anime datasets. The qualitative and quantitative results show that our method can transfer abstract sketch styles while faithfully conveying the original content.