Scaling bioinformatics applications on HPC.

Area of Science:

• Bioinformatics and Computational Biology
• Genomics and Molecular Biology

Background:

• Exponential growth in sequence databases due to advances in molecular biology and next-generation sequencing.
• Traditional parallelization techniques (threading, MPI) are insufficient for timely processing of large biological sequence datasets using BLAST.
• Need for efficient computational methods to handle massive sequence data.

Purpose of the Study:

• To introduce a novel parallelization method for accelerating BLAST sequence analysis.
• To overcome the limitations of existing parallelization techniques for large-scale sequence data.
• To enable faster and more efficient processing of biological sequences.

Main Methods:

• Development of a 'dual segmentation' method for array job parallelization.
• Modification of BLAST source code to accept the number of records in the original database.
• Implementation of strategies for managing and consolidating numerous partial results from segmented tasks.

Main Results:

• Achieved O(T) theoretical speed-up, where T is the number of array job tasks.
• Reduced BLAST job completion time from 27 days on 40 CPUs to 1 day using 4104 tasks (each on 8 CPUs).
• Demonstrated successful processing of large datasets with significantly reduced wallclock time.

Conclusions:

• Dual segmentation enables massive parallelization, reducing task size, scope, and execution time.
• Offers benefits such as fine-grained checkpointing, increased job submission flexibility, and compatibility with older hardware.
• Provides a cost-effective solution for organizations performing large-scale sequence analysis, enabling more exhaustive research.