clubber: removing the bioinformatics bottleneck in big data analyses

Area of Science:

• Computational Biology
• Bioinformatics
• Genomics

Background:

• High-throughput technologies generate large datasets, shifting research bottlenecks to data analysis.
• Existing bioinformatics tools often struggle with the scale of modern biological data.
• High Performance Computing (HPC) resources are powerful but can be complex to utilize effectively.

Purpose of the Study:

• To introduce Clubber, an automated cluster-load balancing system for optimizing big data analyses in biology.
• To reduce computation times for complex bioinformatics tasks.
• To improve the user-friendliness of HPC environments for biological research.

Main Methods:

• Developed an automated system (Clubber) for load balancing parallel submissions across HPC resources.
• Integrated a plug-and-play framework for reusability of bioinformatics solutions.
• Enabled dynamic scaling of HPC resources, including cloud and heterogeneous environments.
• Provided an interactive web interface and RESTful API for job monitoring and result retrieval.

Main Results:

• Clubber significantly reduces computation times for large-scale biological data analysis.
• Demonstrated efficient processing of 21 metagenomes in 172 minutes.
• Applied Clubber to analyze Deepwater Horizon oil-spill data, assessing environmental recovery.
• Utilized Clubber to analyze CAMI-challenge data, revealing insights into microbiome shifts.

Conclusions:

• Clubber effectively optimizes big data analyses in computational biology.
• The system facilitates the use of HPC resources, making them more accessible.
• Clubber's efficiency is crucial for everyday computational biology workflows and discovery.