DACE: a scalable DP-means algorithm for clustering extremely large sequence data

Area of Science:

• Computational biology
• Bioinformatics
• Metagenomics

Background:

• Next-generation sequencing generates vast amounts of data.
• 16S and 18S rRNA gene sequencing are crucial for microbial diversity profiling.
• Efficient clustering of large sequencing datasets is essential for downstream analysis.

Purpose of the Study:

• To develop a scalable algorithm for clustering extremely large sequencing data.
• To improve the efficiency and accuracy of clustering in metagenomic applications.

Main Methods:

• Implementation of a scalable Dirichlet Process Means (DP-means) algorithm named DACE.
• Utilizing an efficient random projection partition strategy for parallel clustering.

Main Results:

• DACE clusters billions of sequences in hours, significantly outperforming existing methods (6-80x faster).
• Achieved high accuracy in clustering large datasets, including Lake Taihu and Ocean TARA data.
• Successfully clustered human gut microbiome gene catalogs, identifying redundant genes.

Conclusions:

• DACE provides a highly efficient and accurate solution for clustering massive sequencing data.
• The algorithm accelerates metagenomic analysis, enabling deeper insights into microbial communities.
• DACE is publicly available for broader scientific application.