Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images

Area of Science:

• Computer Vision
• Medical Imaging
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) are typically trained on low-resolution images due to hardware memory constraints.
• High-resolution, multi-megapixel images are crucial for accurate disease identification in domains like medical imaging.
• Existing methods limit the input image size for CNN training, hindering analysis of large-scale datasets.

Purpose of the Study:

• To propose a novel method for end-to-end training of CNNs with arbitrary input image sizes.
• To demonstrate the feasibility of training CNNs on large, multi-megapixel images without memory limitations.
• To evaluate the performance improvement of CNNs trained on high-resolution images for medical diagnostic tasks.

Main Methods:

• Developed a novel method to train CNNs on large images by processing smaller image tiles during forward and backward passes.
• Implemented an end-to-end training approach that exploits the locality of CNN operations.
• Validated the method on images up to 66-megapixels (8192x8192), achieving significant memory savings (approx. 50GB per image).

Main Results:

• CNNs trained with the proposed method can effectively extract relevant information from high-resolution images.
• Increasing image resolution significantly benefits CNN performance in medical image analysis tasks.
• Achieved an improved area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for breast cancer metastasis detection (CAMELYON17).
• Reached a Spearman correlation metric of 0.570 (16MP), approaching state-of-the-art performance on the TUPAC16 dataset (previously 0.485 at 1MP).

Conclusions:

• The proposed tiling method enables direct end-to-end training of CNNs on large-resolution images, overcoming hardware memory constraints.
• Training with higher resolution images leads to improved diagnostic accuracy in medical imaging applications.
• The method offers a practical solution for leveraging high-resolution data in CNN-based medical image analysis, with code available for reproducibility.