MetaSpark: a spark-based distributed processing tool to recruit metagenomic reads to reference genomes

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Next-generation sequencing generates massive metagenomic data, challenging traditional bioinformatics tools.
• A bottleneck in metagenomic studies is the lack of scalable, cloud-computing-friendly data analysis tools.
• Efficient recruitment of metagenomic reads to reference genomes is crucial for understanding microbial communities.

Purpose of the Study:

• To introduce MetaSpark, a novel Apache Spark-based tool for large-scale metagenomic read recruitment.
• To address the limitations of existing tools in handling massive datasets and cloud environments.
• To improve the efficiency and accuracy of mapping sequencing reads to reference genomes.

Main Methods:

• Developed MetaSpark utilizing Apache Spark's resilient distributed datasets (RDDs) for in-memory caching and distributed processing.
• Implemented a read recruitment algorithm optimized for parallel processing on cluster nodes.
• Evaluated MetaSpark's performance against established tools like SOAP2, BWA, LAST, and FR-HIT using diverse test cases.

Main Results:

• MetaSpark demonstrated significantly higher read recruitment rates compared to SOAP2, BWA, and LAST.
• Achieved approximately 4% increase in recruited reads over FR-HIT with 1 million reads and 0.75 GB of reference data.
• Showcased excellent scalability and high performance across various test scenarios, confirming its suitability for large datasets.

Conclusions:

• MetaSpark offers a scalable and high-performance solution for metagenomic read recruitment, overcoming limitations of traditional tools.
• The tool's ability to leverage distributed computing makes it well-suited for cloud-based analysis of massive datasets.
• MetaSpark represents a significant advancement in bioinformatics tools for analyzing complex metagenomic data.