Robustness of SAM: Segment Anything under Corruptions and beyond

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• The Segment Anything Model (SAM) excels at object segmentation with zero-shot capabilities.
• SAM's robustness to various corruptions and attacks is not well-understood.
• Previous research suggests SAM may prioritize texture over shape.

Purpose of the Study:

• To comprehensively evaluate the robustness of the Segment Anything Model (SAM) under diverse corruptions and adversarial attacks.
• To identify SAM's vulnerabilities and the underlying mechanisms causing them.
• To explore defense strategies and improve adversarial transferability for SAM.

Main Methods:

• Evaluated SAM's performance against synthetic corruptions (style transfer) and 15 common corruptions.
• Assessed resilience to local patch occlusion and adversarial patch attacks.
• Investigated robustness against imperceptible global adversarial attacks using basic attacks and a mask-aware loss.
• Explored contrastive learning for enhanced adversarial example transferability across SAM models.

Main Results:

• SAM demonstrated robustness against synthetic and common corruptions.
• SAM was resilient to local patch occlusion but vulnerable to adversarial patch attacks.
• Adversarial attacks disrupt SAM's feature maps and attention mechanisms.
• SAM's vulnerability is linked to its reliance on high-frequency signals.

Conclusions:

• SAM exhibits notable robustness in many scenarios but is susceptible to specific adversarial attacks.
• Understanding SAM's reliance on signal frequencies is key to improving its robustness.
• Further research into defense mechanisms and adaptive attacks is crucial for real-world SAM deployment.