CTFS: A consolidated transformer framework for instance and semantic segmentation tasks.

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• Instance and semantic segmentation are key computer vision tasks.
• Existing unified transformer frameworks struggle with concurrent multi-task optimization and gradient conflicts.

Purpose of the Study:

• To develop a consolidated Transformer framework (CTFS) for efficient simultaneous instance and semantic segmentation.
• To address challenges in optimizing shared parameters and mitigating gradient conflicts in multi-task learning.

Main Methods:

• Introduced an affinity-guided sharing strategy (AGSS) for staged learning of task-shared parameter proportions and distributions.
• Proposed a fine-grained gradient rectification strategy (FGRS) to resolve element-wise gradient conflicts during backpropagation.
• Built the CTFS framework upon the standard Swin Transformer architecture.

Main Results:

• CTFS achieved impressive performance on instance segmentation (COCO dataset) and semantic segmentation (ADE20K dataset).
• The AGSS strategy reduced the difficulty of network optimization by using shared parameter proportions as prior knowledge.
• The FGRS strategy effectively mitigated gradient conflicts at the element level.

Conclusions:

• The consolidated Transformer framework (CTFS) offers an effective solution for simultaneous instance and semantic segmentation.
• The proposed AGSS and FGRS strategies significantly improve network optimization and gradient conflict resolution.
• CTFS demonstrates strong performance without complicating the underlying Swin Transformer architecture.