clustermq enables efficient parallelization of genomic analyses.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• High performance computing (HPC) clusters are essential for large-scale bioinformatics analysis and modeling.
• Existing R packages for submitting analyses to HPC schedulers exhibit poor scalability and high processing overhead for numerous tasks.

Purpose of the Study:

• To introduce clustermq, an R package designed to overcome the limitations of existing tools for parallelizable workflows on HPC clusters.
• To demonstrate the performance improvements offered by clustermq for large-scale bioinformatics tasks.

Main Methods:

• Development of the clustermq R package for efficient job submission and management on HPC schedulers.
• Performance benchmarking of clustermq against previously available R packages for HPC task parallelization.

Main Results:

• clustermq processes analyses up to three orders of magnitude (1000x) faster than alternative R packages.
• The package effectively addresses the scalability bottleneck associated with high numbers of tasks in bioinformatics analyses.
• Demonstrated application in investigating genomic associations of drug sensitivity in cancer cell lines.

Conclusions:

• clustermq offers a substantial performance improvement for parallelizable workflows in bioinformatics.
• The package provides a scalable and efficient solution for leveraging HPC resources in R.
• clustermq is readily available on CRAN and GitHub for broad adoption.