SpaRC: scalable sequence clustering using Apache Spark

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Next-generation sequencing (NGS) studies generate vast amounts of data (100-1000 GB).
• Assembling large-scale transcriptomic and metagenomic datasets presents a challenge, requiring a balance between scalability and accuracy.
• Existing assembly methods often compromise accuracy for scalability, or vice versa.

Purpose of the Study:

• To develop a scalable application for partitioning NGS reads based on their molecule of origin.
• To enable downstream assembly optimization for large-scale transcriptomics and metagenomics data.
• To provide a solution that balances scalability and accuracy in sequence data assembly.

Main Methods:

• Developed an Apache Spark-based scalable sequence clustering application named SparkReadClust (SpaRC).
• SpaRC partitions sequencing reads to facilitate optimized downstream assembly.
• The application was tested on both short and long read sequencing technologies for transcriptomes and metagenomes.

Main Results:

• SpaRC demonstrated high clustering performance on diverse NGS datasets.
• The application achieved near-linear scalability with increasing data size and compute nodes.
• SpaRC performed comparably on cloud computing and High-Performance Computing (HPC) environments.

Conclusions:

• SpaRC offers a scalable solution for clustering billions of reads from NGS experiments.
• Apache Spark provides a cost-effective platform for rapid development and deployment of large-scale sequence data analysis tools.
• The developed tool addresses the critical need for accurate and scalable assembly of complex genomic datasets.