SparkGA2: Production-quality memory-efficient Apache Spark based genome analysis framework.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Decreasing costs of Next Generation Sequencing (NGS) drive increased use for genetic disease diagnosis.
• NGS generates massive datasets (hundreds of gigabytes per experiment), necessitating efficient and scalable analysis tools.

Purpose of the Study:

• Introduce SparkGA2, a production-quality, memory-efficient framework for high-performance DNA analysis in the cloud.
• Enable scalable analysis by adjusting computational resources (nodes) and optimizing memory usage.

Main Methods:

• Utilizes Apache Spark's in-memory caching for accelerated processing.
• Implements on-the-fly compression of intermediate data to reduce memory footprint by up to 3x.
• Employs a streaming approach for gradual input data processing.

Main Results:

• SparkGA2 achieves up to 22% speedup on a 67-node cluster and 9% on a 6-node cluster compared to state-of-the-art methods.
• Incorporating data pre-processing via streaming, SparkGA2 shows a 51% speedup on a 6-node cluster.
• The framework adapts to systems with lower memory, offering flexibility with minor performance trade-offs.

Conclusions:

• SparkGA2 provides a faster and more memory-efficient solution for analyzing large-scale NGS data.
• The framework's scalability and adaptability make it suitable for diverse cloud computing environments.
• Publicly available source code facilitates adoption and further development in genomic data analysis.