Hierarchical AttentionShift for Pointly Supervised Instance Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Pointly supervised instance segmentation (PSIS) faces challenges due to appearance variations causing semantic inconsistency.
• Existing methods struggle to capture fine-grained object details and semantic relationships effectively.

Purpose of the Study:

• To propose a novel Hierarchical AttentionShift approach to resolve semantic inconsistency in PSIS.
• To exploit hierarchical semantics and key-point representations for improved object understanding.
• To enhance the self-attention mechanism for fine-grained vision tasks.

Main Methods:

• Developed a hierarchical AttentionShift approach operating at instance, part, and fine-grained levels.
• Utilized iterative spatial and feature-space estimation of representative key points.
• Transformed conventional self-attention into hierarchical activation with local refinement.

Main Results:

• Achieved significant improvements on PASCAL VOC 2012 Aug and MS-COCO 2017 benchmarks, outperforming state-of-the-art (SOTA) methods.
• Demonstrated a 10.4% and 7.0% increase in mean average precision (mAP)₅₀ on the respective benchmarks.
• Improved the Segment Anything Model (SAM) by 9.4% AP on the COCO test-dev dataset.

Conclusions:

• Hierarchical AttentionShift effectively addresses semantic inconsistency in PSIS by leveraging hierarchical semantics and key-point representations.
• The proposed method offers a new perspective for regularizing self-attention in fine-grained vision tasks.
• The approach shows strong performance gains and broad applicability, including integration with large foundation models like SAM.