DiffRES: Unleashing Text-to-Image Diffusion Models for Generative Referring Expression Segmentation Without...

Referring Expression Segmentation (RES) requires models not only to locate objects specified by referring expressions accurately but also to predict complete and precise masks. Existing methods primarily focus on complex multimodal alignment for object grounding, often neglecting mask quality, which results in incomplete foreground regions and imprecise boundaries. To address these challenges, we propose DiffRES, a mask-generating framework based on Stable Diffusion (SD), designed to tackle the RES problem with a focus on achieving high-quality masks. DiffRES effectively mitigates the information leakage issue prevalent in existing generative dense prediction diffusion models, which allows the model to infer the target's position directly from noisy masks during training without understanding the text condition, leading to severe overfitting. Specifically, DiffRES directly guides SD with visual and linguistic information to generate target binary masks, fundamentally bypassing the information leakage issue. This approach enables efficient knowledge transfer from SD to the RES task, resulting in precisely localized binary masks with sharp and precise boundaries. Extensive experiments show that DiffRES surpasses current state-of-the-art traditional methods on boundary precision ( $AP_{b}$ ) which is sensitive to mask quality, while also significantly outperforming all existing SD-based RES models across all metrics. Our code is publicly available at https://github.com/charon517-517/DiffRES.