VC@Scale: Scalable and high-performance variant calling on cluster environments.

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Deep learning methods like DeepVariant offer superior accuracy in variant calling compared to traditional algorithms.
• Existing workflows for deep learning variant calling are computationally expensive and not highly scalable.
• Current cluster-scaled workflows often fail to fully leverage Apache Spark's in-memory processing capabilities.

Purpose of the Study:

• To develop a scalable and high-performance workflow for deep learning-based variant calling.
• To optimize the integration of pre-processing and variant-calling stages within a single framework.
• To enhance the efficiency of next-generation sequencing data analysis on high-performance computing clusters.

Main Methods:

• Developed a native Apache Spark-based workflow for variant calling.
• Utilized Python and Apache Arrow for efficient in-memory data transfer and transformation.
• Integrated pre-processing stages using Spark's built-in functions for sorting and duplicate marking.

Main Results:

• Achieved over 2x performance improvement in pre-processing stages compared to state-of-the-art methods.
• Demonstrated a scalable and high-performance solution for DeepVariant on both CPU and CPU+GPU clusters.
• Showcased efficient resource utilization for variant-calling analysis.

Conclusions:

• The proposed approach enables feasible and easy scalability for high-performance variant-calling analysis.
• The workflow leverages standardized Apache Arrow data representations for efficient processing.
• All implementation codes and configurations are publicly available and open-sourced.