Efficient and accurate search in petabase-scale sequence repositories

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Public biological sequencing data is rapidly expanding, posing challenges for efficient full-text searching.
• Existing methods struggle to handle the scale and complexity of large sequence datasets.

Purpose of the Study:

• To present MetaGraph, a scalable framework for indexing and full-text searching of biological sequence data.
• To demonstrate the cost-effectiveness and feasibility of searching massive sequence repositories.

Main Methods:

• Utilizing annotated de Bruijn graphs for scalable indexing of DNA, RNA, and protein sequences.
• Integrating data from seven public sources, encompassing 18.8 million unique sequence sets and 210 billion amino acid residues.
• Developing a cost-effective search mechanism for large sequence repositories.

Main Results:

• Achieved full-text searchability for 18.8 million unique DNA/RNA sequence sets and 210 billion amino acid residues.
• Demonstrated cost-effective searching of 67 petabase pairs of raw sequence data, with costs as low as $0.74 per queried megabase pair.
• Showcased that a compressed representation of all public biological sequences can fit on consumer hard drives for approximately $2,500.

Conclusions:

• MetaGraph provides a cost-effective and scalable solution for full-text searching of biological sequence data.
• The framework facilitates integrative analyses and has the potential to accelerate biomedical research advancements.
• Accessible and searchable large-scale biological data can catalyze new discoveries and applications.