PipeMEM: A Framework to Speed Up BWA-MEM in Spark with Low Overhead.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• DNA sequence alignment is crucial for genome analysis.
• BWA-MEM is a fast, accurate single-node aligner, but struggles with large datasets.
• Existing Spark-based solutions for BWA-MEM alignment incur high overhead.

Purpose of the Study:

• To present PipeMEM, a novel framework to accelerate BWA-MEM.
• To reduce overhead in Spark-based BWA-MEM alignment.
• To enhance genome data processing capabilities.

Main Methods:

• Developed PipeMEM, a framework leveraging Spark's pipe operation.
• Implemented a pipeline structure within PipeMEM.
• Utilized in-memory computation to further accelerate processing.

Main Results:

• PipeMEM demonstrated low overhead in paired-end alignment tasks.
• Achieved 2.27x speedup compared to BWASpark on multi-node systems.
• Achieved 2.33x speedup compared to SparkBWA on multi-node systems.

Conclusions:

• PipeMEM effectively accelerates BWA-MEM in a Spark environment.
• The framework offers high performance with reduced overhead.
• PipeMEM provides an efficient solution for large-scale genome alignment challenges.