GradToken: Decoupling tokens with class-aware gradient for visual explanation of Transformer network

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Transformer networks are increasingly utilized across diverse AI fields, including computer vision and natural language processing.
• Explaining Transformer decision-making is crucial for enhancing trust and understanding in real-world applications.
• Existing explanation methods for convolutional neural networks are not directly applicable to Transformer architectures due to structural differences.

Purpose of the Study:

• To address the challenge of designing effective explanation methods specific to Transformer networks.
• To overcome the semantic coupling problem in Transformer attention weight matrices that hinders category-specific explanations.
• To propose a novel method for generating visual explanations of Transformer predictions.

Main Methods:

• Analyzed the semantic coupling issue within Transformer attention weight matrices.
• Proposed GradToken, a gradient-decoupling-based token relevance method.
• Utilized class-aware gradients to decouple semantics and leveraged token relations to generate relevance maps.

Main Results:

• GradToken effectively decouples tangled semantics in class tokens, enabling category-specific explanations.
• Generated relevance maps accurately highlight target regions, improving visual explanation quality.
• Extensive experiments validated the method's effectiveness and reliability.

Conclusions:

• GradToken provides a specific and effective solution for explaining Transformer predictions visually.
• The method enhances the interpretability of Transformers by generating focused and reliable relevance maps.
• This work contributes to building greater trust and understanding in the application of Transformer models.