Learn to explain transformer via interpretation path by reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Transformer models are crucial in AI but their complexity hinders interpretation.
• Existing methods often focus on single internal variables, limiting comprehensive understanding.
• Understanding Transformer decision-making is vital for trust and robustness.

Purpose of the Study:

• To develop a unified framework for interpreting Transformer models using multiple internal variables.
• To enhance the effectiveness of model interpretation and adversarial attack strategies.
• To compare the interpretability contributions of different internal Transformer features.

Main Methods:

• Introduced a Reinforcement Learning environment for step-by-step input sequence modification.
• The agent learns targeted token modification strategies to reduce model confidence.
• The agent can utilize single or combined internal variables (attention matrices, gradients, hidden states, activations) as observations.

Main Results:

• Demonstrated superior performance in model interpretation and adversarial attack tasks across three real-world datasets.
• The proposed method significantly improves interpretation effectiveness compared to random sampling.
• Enabled comparison of the interpretability contributions of various internal Transformer variables.

Conclusions:

• The Reinforcement Learning environment offers a powerful and flexible approach to Transformer interpretability.
• Findings provide insights into Transformer decision-making and inspire future research.
• The unified model interpretation framework advances the field of explainable AI.