CLIP Graph Adaptor: A Dual-Graph Adapted Visual-Language Model for Weakly Supervised Semantic Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Weakly supervised semantic segmentation (WSSS) utilizes contrastive language-image pretraining (CLIP) for pseudo-label generation.
• Direct CLIP application in WSSS leads to suboptimal transferability and generalization due to ignored interclass relationships.

Purpose of the Study:

• To propose CLIP Graph Adapter (CLIP-GA), a novel approach for generating high-quality initial class activation maps (CAMs).
• To enhance WSSS performance by integrating textual and visual structural knowledge, addressing limitations of direct CLIP model application.

Main Methods:

• Introduced a dual-graph adaptive strategy with textual and visual subgraphs.
• Employed cross-modal graph attention (CGA) for effective knowledge fusion.
• Utilized specialized loss functions, superpixel consistency, and graph reasoning attention (GRA) for pseudo-label refinement and global contextual understanding.

Main Results:

• CLIP-GA effectively generates high-quality initial CAMs by integrating structural knowledge.
• The approach demonstrated superior performance compared to state-of-the-art methods on PASCAL VOC 2012 and MS COCO 2014 datasets.
• Achieved improved object region completeness and reduced background activation.

Conclusions:

• CLIP-GA significantly advances WSSS by effectively leveraging multimodal structural information.
• The proposed method offers improved transferability and generalization for semantic segmentation tasks.
• The integration of graph-based strategies enhances the quality of pseudo-labels and final segmentation outcomes.