Object-based feedback attention in convolutional neural networks improves tumour detection in digital pathology

Human visual attention allows prior knowledge or expectations to influence visual processing, allocating limited computational resources to only that part of the image that are likely to behaviourally important. Here, we present an image recognition system based on biological vision that guides attention to more informative locations within a larger parent image, using a sequence of saccade-like motions. We demonstrate that at the end of the saccade sequence the system has an improved classification ability compared to the convolutional neural network (CNN) that represents the feedforward part of the model. Feedback activations highlight salient image features supporting the explainability of the classification. Our attention model deviates substantially from more common feedforward attention mechanisms, which linearly reweight part of the input. This model uses several passes of feedforward and backward activation, which interact non-linearly. We apply our feedback architecture to histopathology patch images, demonstrating a 3.5% improvement in accuracy (p < 0.001) when retrospectively processing 59,057 9-class patches from 689 colorectal cancer WSIs. In the saccade implementation, overall agreement between expert-labelled patches and model prediction reached 93.23% for tumour tissue, surpassing inter-pathologist agreement. Our method is adaptable to other areas of science which rely on the analysis of extremely large-scale images.