DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Weakly Supervised Semantic Segmentation (WSSS) commonly uses Class Activation Maps (CAMs) for pixel-level predictions from image-level labels.
• Contrastive Language-Image Pre-training (CLIP) has emerged for CAM generation in WSSS, but often suffers from limited dense knowledge in visual and text modalities, leading to suboptimal CAMs.

Purpose of the Study:

• To introduce DiCLIP, a novel WSSS framework that enhances CLIP's dense knowledge using generative diffusion models.
• To address the limitations of existing WSSS methods by improving spatial awareness and semantic representation.

Main Methods:

• Proposed Visual Correlation Enhancement (VCE) module with Attention Clustering Refinement (ACR) to improve spatial awareness and mitigate over-smoothing in CLIP's attention.
• Introduced Text Semantic Augmentation (TSA) module leveraging diffusion models for a dynamic key-value cache, shifting to a visual knowledge retrieval paradigm for richer text semantics.

Main Results:

• DiCLIP significantly outperforms state-of-the-art methods on benchmark datasets like PASCAL VOC and MS COCO.
• The proposed framework demonstrates a notable reduction in training costs compared to existing approaches.

Conclusions:

• DiCLIP effectively leverages generative diffusion models to enhance CLIP for improved WSSS performance.
• The framework offers a more efficient and effective solution for dense prediction tasks in computer vision.