SalientNet: mechanistic and reliable saliency maps for medical image classification via unsupervised learning

Interpreting neural network decisions is crucial in healthcare, where both global and local validations are essential for building trust. While saliency maps offer intuitive visualizations of model attention, their limitations-including low sensitivity to input perturbations, lack of causal grounding, and limited mechanistic insight-reduce their reliability in critical medical scenarios. This study aims to develop a more trustworthy and mechanistically interpretable framework for explaining convolutional neural network (CNN) decisions in medical image classification.

We propose SalientNet, a data- and architecture-agnostic unsupervised neural network framework for interpreting CNNs. SalientNet consists of two jointly trained components: a CNN examiner, which uses a novel variable channel convolution (VCC) module to extract architecture-independent information from a CNN's intermediate feature maps, and an attribution map generator, built from two coupled U-Nets, which combines this information with the input image to produce a saliency map. Training requires no ground-truth saliency annotations; instead, an Interpretation Loss enforces that the attribution map retains just enough information for a frozen classifier to reproduce its original prediction within a controlled tolerance (dissimilarity temperature), preventing both trivial and overly aggressive masks. We evaluated SalientNet on three medical imaging datasets spanning different modalities and body regions-BreakHis (histopathology), Brain MRI ND5, and FracAtlas (musculoskeletal X-rays)-using ResNet-18, MobileNetV3-Small, and EfficientNet-B0 as the interpreted classifiers.

Through extensive experiments, we demonstrate that SalientNet outperforms popular saliency-based benchmarks, including Grad-CAM, Grad-CAM++, and LIME, achieving substantially higher faithfulness (Win %) and lower prediction Drop (%) across nearly all dataset-architecture combinations, while keeping spurious confidence increases (Inc %) comparably low. On FracAtlas, SalientNet's explanations also showed strong spatial correlation with fracture regions annotated by medical practitioners, with a mean ground-truth active fraction of up to 0.96 for ResNet-18. A mechanistic analysis further shows that nearly all CNN layers contribute meaningfully to SalientNet's explanations, supporting its interpretive fidelity.

Our unsupervised framework provides improved interpretability for CNNs in healthcare applications, offering a more reliable and mechanistically grounded alternative to traditional saliency methods for local validation and building trust in AI-assisted medical diagnosis. Our implementation is publicly available https://github.com/S-iddhaarth/Unsupervised-Training-of-an-Interpretable-Neural-Network-to-Explain-Convolutional-Neural-Networks here.