Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep learning models are widely used but their decision-making processes are often opaque.
• Existing interpretation methods for neural networks typically focus on individual components, missing a holistic view.
• Understanding learned features and their interactions is crucial for reliable AI systems.

Purpose of the Study:

• To develop a scalable and systematic method for interpreting deep learning models.
• To visualize what features deep learning models learn and how they interact to produce predictions.
• To provide insights into complex neural network representations and inform architecture design.

Main Methods:

• Introduced Summit, an interactive system for summarizing and visualizing deep learning model features.
• Developed two scalable summarization techniques: activation aggregation and neuron-influence aggregation.
• Created a novel attribution graph to reveal crucial neuron associations and substructures.

Main Results:

• Summit effectively scales to large datasets like ImageNet (1.2M images).
• The attribution graph summarizes complex neural network behaviors into compact, interactive visualizations.
• Exploration scenarios revealed surprising insights into image classifier representations.

Conclusions:

• Summit offers a powerful tool for understanding and debugging large-scale deep learning models.
• The system aids in discovering learned representations and guiding future neural network architecture improvements.
• Summit is open-source and accessible via modern web browsers.