Rectify ViT Shortcut Learning by Visual Saliency.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Shortcut learning in deep learning models leads to poor generalization and interpretability.
• Vision Transformers (ViT) are widely used but their shortcut learning behavior is not well understood.
• Domain-specific knowledge, like eye-gaze data, can guide models but is often impractical to obtain.

Purpose of the Study:

• To propose a novel Saliency-guided ViT (SGT) model to rectify shortcut learning in ViT.
• To leverage human prior knowledge via visual saliency without requiring eye-gaze data.
• To improve the generalizability and interpretability of ViT models.

Main Methods:

• A computational visual saliency model generates saliency maps for input images.
• Saliency maps filter informative image patches, focusing the model on relevant regions.
• A residual connection with self-attention across all patches mitigates global information loss.

Main Results:

• The SGT framework effectively learns and applies human prior knowledge without eye-gaze data.
• Achieved significantly better performance compared to baseline models on natural and medical image datasets.
• Successfully rectified harmful shortcut learning, enhancing ViT model interpretability.

Conclusions:

• The SGT model demonstrates an effective method for rectifying shortcut learning in ViT.
• Transferring human prior knowledge through visual saliency is a promising approach.
• The SGT framework offers improved performance, generalizability, and interpretability for ViT models.