Optimising parallel R correlation matrix calculations on gene expression data using MapReduce.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• High-throughput molecular profiling generates large datasets crucial for clinical decision-making through subject stratification.
• Unsupervised clustering algorithms are used for stratification, but their speed is limited by inefficient correlation matrix calculations.
• The increasing scale of molecular data necessitates optimized algorithms to maintain performance.

Purpose of the Study:

• To evaluate current parallel correlation calculation methods.
• To introduce an efficient MapReduce-based algorithm for optimizing correlation calculations in large-scale molecular data analysis.
• To improve the speed and scalability of analyzing high-throughput genomic data.

Main Methods:

• Developed a data distribution and parallel calculation algorithm using the MapReduce framework.
• Implemented the algorithm using the R package RHIPE.
• Evaluated performance using micro- and macro-benchmarks with gene expression data and the TCGA dataset.

Main Results:

• The MapReduce-based RHIPE implementation showed significant speedups (3.26-5.83x) over default Snowfall and basic RHIPE for Euclidean, Pearson, and Spearman correlations.
• In macro-benchmarks, optimized RHIPE was 2.03-16.56x faster than vanilla R and 1.22-1.71x faster than optimized Snowfall, including faster data preparation.
• Both optimized RHIPE and Snowfall completed Kendall correlation on the TCGA dataset within 7 hours, over 30x faster than estimated vanilla R.

Conclusions:

• The MapReduce algorithm implemented in RHIPE outperforms vanilla R and conventional parallel algorithms like R Snowfall for molecular data analysis.
• The MapReduce framework is highly promising for analyzing large, high-dimensional genomic datasets.
• This algorithm serves as a foundation for optimizing correlation calculations in Big Data from high-throughput molecular profiling.