Related Experiment Video
Updated: Feb 26, 2026
07:05
Applying Hyperspectral Reflectance Imaging to Investigate the Palettes and the Techniques of Painters
Published on: June 18, 2021
2.9K
Theoretical and Empirical Comparison of Big Data Image Processing with Apache Hadoop and Sun Grid Engine
Shunxing Bao1, Frederick D Weitendorf1, Andrew J Plassard1
1Computer Science, Vanderbilt University, Nashville, TN, USA 37235.
Summary
Big data frameworks like Hadoop offer advantages for medical imaging by co-locating data and computation, minimizing network bottlenecks. This study evaluates when these big data solutions are most relevant compared to traditional cluster computing.
Area of Science:
- Medical Imaging
- Computational Science
- Big Data Analytics
Background:
- Traditional medical imaging processing relies on cluster computers and network file systems (NFS), which can face performance issues with large datasets due to network bandwidth saturation.
- The emergence of big data technologies like Hadoop and HBase offers a potential solution by enabling co-location of data storage and computation, thereby minimizing data transfer.
Purpose of the Study:
- To formally evaluate the benefits of Hadoop-based frameworks against traditional approaches in medical imaging.
- To characterize the transition point from large-scale processing to big data challenges that necessitate alternative computational frameworks.
- To provide a comparative analysis of Hadoop relevance for medical imaging workloads.
Main Methods:
- Implemented a Hadoop system on a production lab-cluster alongside a standard Sun Grid Engine (SGE).
- Developed and validated theoretical models for wall-clock time and resource time for both Hadoop and traditional approaches.
- Empirically assessed computational performance using three T1 image archives across cluster hardware configurations (72, 109, 209 CPU cores) and varying job lengths.
Main Results:
- Empirical results demonstrated a strong match with the developed theoretical models.
- Comparative analysis based on empirical data provides insights into the performance trade-offs between Hadoop and traditional methods.
- The study identifies specific conditions under which Hadoop frameworks offer significant advantages for medical imaging.
Conclusions:
- Hadoop frameworks are relevant for medical imaging when dealing with large datasets and frequent data access, where traditional NFS approaches become a bottleneck.
- The study provides a data-driven framework for selecting the appropriate computational approach based on dataset size, job length, and available hardware.
- This research contributes to optimizing computational strategies in medical imaging for efficient big data processing.
