DGADiff: Decoupled Guide Attention with Diffusion Model for Portrait Stylization

Diffusion-based models have substantially propelled the progress of portrait stylization. Nevertheless, the lack of clear supervisory signals often leads to pattern drift in the target portrait. To overcome this issue, we introduce DGADiff, a training-free stylization framework based on a diffusion model. Specifically, we first leverage prior knowledge from a pre-trained latent consistency model (LCM) to efficiently sample representative features from noisy image pairs. Next, we design a Decoupled Guide Attention Mechanism (DGA), that disentangles the U-Net attention into separate self-attention and masked-attention tracks, enabling accurate transfer of fine-grained facial style patterns. Extensive experiments verify that our DGADiff achieves favorable results across multiple metrics in content-to-style and style-to-content multi-domain tasks, demonstrating the effectiveness of spatial attention decoupling for portrait stylization.