HPC-CLUST: distributed hierarchical clustering for large sets of nucleotide sequences

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• The exponential growth of nucleotide sequence data necessitates efficient analysis methods.
• Sequence clustering is crucial for reducing redundancy, identifying gene families, and defining taxonomic units.
• Traditional clustering algorithms face scalability challenges in terms of time and memory.

Purpose of the Study:

• To present HPC-CLUST, a software pipeline for clustering large sets of pre-aligned DNA sequences.
• To enable efficient analysis of massive sequence datasets using distributed computing hardware.
• To provide a scalable solution that overcomes the limitations of exact clustering algorithms.

Main Methods:

• Development of HPC-CLUST, a highly optimized software pipeline.
• Implementation in C++ utilizing the Message Passing Interface (MPI) standard for distributed computing.
• Efficient allocation of memory and computing resources for parallel processing.

Main Results:

• HPC-CLUST can cluster large numbers of pre-aligned DNA sequences on distributed hardware.
• The pipeline demonstrates efficient memory and computing resource allocation.
• It successfully processes over a million sequences within a few hours on a small cluster.

Conclusions:

• HPC-CLUST offers a scalable and efficient solution for clustering large DNA sequence datasets.
• The software pipeline overcomes the computational bottlenecks associated with traditional clustering methods.
• Freely available source code and binaries facilitate widespread adoption in bioinformatics research.