Causal Interpretation of Neural Network Computations with Contribution Decomposition

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Understanding neural network internal workings is vital for interpretation and manipulation.
• Current methods often focus on activation patterns, limiting causal insights.
• Analyzing how hidden neurons directly influence outputs is key to deeper comprehension.

Purpose of the Study:

• To introduce CODEC (Contribution Decomposition), a novel method for analyzing neural network behavior.
• To reveal causal processes within neural networks that are not apparent from activation analysis alone.
• To enhance the interpretability and controllability of artificial neural networks.

Main Methods:

• Utilized sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions.
• Applied CODEC to benchmark image-classification networks and models of vertebrate retinal neural activity.
• Focused on analyzing the direct contributions of neurons to network outputs, rather than just activations.

Main Results:

• Contributions increase in sparsity and dimensionality across network layers.
• Positive and negative contributions to network outputs progressively decorrelate.
• CODEC enabled causal manipulation of network outputs and interpretable visualizations of image components.
• Uncovered combinatorial actions of interneurons and identified sources of dynamic receptive fields in retinal models.

Conclusions:

• CODEC provides a powerful framework for understanding nonlinear computations across hierarchical layers.
• Contribution modes serve as an informative unit for mechanistic insights into artificial neural networks.
• The method offers enhanced interpretability and control over intermediate network layers.