BAMSI: a multi-cloud service for scalable distributed filtering of massive genome data

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Next-generation sequencing (NGS) generates large whole-genome datasets, posing challenges for analysis.
• Curated variant calls are insufficient for some scientific questions, necessitating direct analysis of raw or aligned reads.
• The sheer size of genomic datasets makes large-scale re-analysis prohibitively expensive without specialized computing facilities.

Purpose of the Study:

• To develop a scalable, storage-agnostic framework for filtering large genomic datasets.
• To enable custom filtering of genomic data without requiring access to large-scale computing infrastructure.
• To make re-analysis of large genomic datasets feasible and cost-effective.

Main Methods:

• Developed a scalable, storage-agnostic framework with an associated API and web UI for custom filtering.
• Implemented a Software as a Service (SaaS) solution named BAMSI for filtering the 1000 Genomes phase 3 dataset.
• Utilized multiple data mirrors and deployed filtering workers close to data in cloud environments for enhanced scalability.

Main Results:

• BAMSI demonstrates improved horizontal scalability by deploying filtering workers near data sources in cloud environments.
• The framework successfully filtered the 1000 Genomes phase 3 dataset, enabling analysis of structural variations.
• The filtering process significantly reduces data size for subsequent analysis.

Conclusions:

• BAMSI provides a flexible framework for efficient filtering of large genomic datasets, optimizing compute and storage resource utilization.
• The framework facilitates downstream interactive analysis by reducing data size and enabling integration with tools like Hadoop, Hive, and Spark.
• BAMSI offers a model for cost-effective hosting and accessible analysis of large, high-value genomic datasets.