MGA-CLIP: A Multigranularity Attribution Framework for Cross-Modal Explainability in CLIP.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Contrastive Language-Image Pretraining (CLIP) models excel in multimodal tasks but lack interpretability.
• Existing visual explanation methods for CLIP are often unstable and imprecise due to reliance on single-gradient information.

Purpose of the Study:

• To develop a novel multigranularity attribution framework for CLIP (MGA-CLIP) to enhance the interpretability of deep multimodal neural networks.
• To provide systematic analysis of internal learning behaviors and cross-modal alignment mechanisms in CLIP.

Main Methods:

• Propose MGA-CLIP, a framework combining coarse-grained integrated gradients (IGs) for stable global interpretation and fine-grained analysis leveraging cross-modal alignment and attention.
• Establish an explicit linkage between channel-level global attributions and patch-level semantic reasoning for consistent explanations across granularities.
• Fuse coarse- and fine-grained results to generate semantically consistent and cross-modally aligned explanations.

Main Results:

• MGA-CLIP demonstrates strong capability in revealing CLIP's internal reasoning during text-based adversarial patch experiments.
• The framework achieves superior attribution stability and semantic focus compared to existing methods.
• Extensive experiments validate the effectiveness of MGA-CLIP in enhancing the interpretability of deep multimodal neural networks.

Conclusions:

• MGA-CLIP provides a robust and effective solution for interpreting the complex mechanisms of CLIP models.
• The proposed framework advances the field of explainable AI for multimodal deep learning.
• MGA-CLIP enables more insightful analysis of multimodal alignment and reasoning processes.