Parallel Versus Distributed Data Access for Gigapixel-Resolution Histology Images: Challenges and Opportunities

Area of Science:

• Digital Pathology
• Computational Pathology
• Medical Imaging

Background:

• Whole-slide imaging (WSI) offers high-resolution pathology images, exceeding gigabytes.
• WSI is increasingly used in routine workflows by leading institutions.
• Large file sizes and proprietary formats hinder WSI data accessibility and high-throughput analysis.

Purpose of the Study:

• To develop novel scalable access methods for handling large WSI files.
• To overcome limitations of traditional preprocessing approaches for WSI data.
• To enable dynamic adjustment of data scalability and unit sizes for efficient processing.

Main Methods:

• Development of novel scalable access methods for parallel file systems.
• Implementation of methods for distributed file/object storage systems.
• Experimental validation using Lustre parallel file system and AWS S3.

Main Results:

• Demonstrated tangible scalability and high-throughput advantages over traditional methods.
• Showcased opportunities not realizable with conventional WSI data handling.
• Validated the effectiveness of the novel methods on parallel and distributed storage systems.

Conclusions:

• Novel scalable access methods significantly enhance WSI data processing efficiency.
• These methods address challenges posed by large file sizes and proprietary formats in digital pathology.
• The approach offers a promising solution for high-throughput analysis in computational pathology workflows.