Optimized distributed systems achieve significant performance improvement on sorted merging of massive VCF files

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Sorted merging of genomic data is crucial for sequencing studies.
• Large-scale projects require merging numerous variant call format (VCF) files.
• Traditional methods face computational and I/O bottlenecks with large datasets.

Purpose of the Study:

• To design and optimize schemas for distributed big data platforms to efficiently merge large VCF files.
• To leverage Apache Hadoop (MapReduce), HBase, and Spark for scalable genomic data merging.

Main Methods:

• Developed custom, optimized schemas for Hadoop, HBase, and Spark using a divide-and-conquer strategy.
• Implemented sequential phases and parallel subtasks to avoid bottlenecks.
• Benchmarked performance against traditional methods, MPI-based HPC, and VCFTools.

Main Results:

• All three developed schemas demonstrated significant efficiency improvements.
• Schemas exhibited superior strong and weak scalability compared to traditional approaches.
• Successful merging of multiple VCF files into single TPED or VCF files was achieved.

Conclusions:

• The proposed schemas offer generalized, scalable solutions for merging genetics and genomics data.
• Apache distributed systems provide a powerful platform for high-performance genomic data processing.
• Optimized workflows are essential for maximizing the benefits of distributed computing in genomics.