Neuron Abandoning Attention Flow: Visual Explanation of Dynamics Inside CNN Models

In this paper, we present a Neuron Abandoning Attention Flow (NAFlow) method to address the unsolved problem of visually explaining the attention evolution dynamics inside CNNs when making their classification decisions. A novel cascading neuron abandoning back-propagation algorithm is designed to precisely exclude the abandoned neurons on all intermediate layers inside a CNN model for the first time. Firstly, a Neuron Abandoning Back-Propagation module is proposed to generate Back-Propagation Feature Maps (BPFM) by using inverse function of the intermediate layers of CNN models, on which the neurons not used for decision-making are removed. Meanwhile, the cascading NA-BP modules calculate the tensors of importance coefficients which are linearly combined with the tensors of BPFMs to form the NAFlow. Secondly, to be able to visualize attention flow for similarity metric-based CNN models, a new channel contribution weights module is proposed to calculate the importance coefficients via Jacobian Matrix. Extensive evaluations demonstrate the effectiveness of the proposed NAFlow across eleven widely-used CNN models for various tasks of general image classification, contrastive learning classification, few-shot image classification, and image retrieval.