Ground truth based comparison of saliency maps algorithms

Deep neural networks (DNNs) have achieved outstanding results in domains such as image processing, computer vision, natural language processing and bioinformatics. In recent years, many methods have been proposed that can provide a visual explanation of decision made by such classifiers. Saliency maps are probably the most popular. However, it is still unclear how to properly interpret saliency maps for a given image and which techniques perform most accurately. This paper presents a methodology to practically evaluate the real effectiveness of saliency map generation methods. We used three state-of-the-art network architectures along with specially prepared benchmark datasets, and we proposed a novel metric to provide a quantitative comparison of the methods. The comparison identified the most reliable techniques and the solutions which usually failed in our tests.