Balancing efficient analysis and storage of quantitative genomics data with the D4 format and d4tools
Hao Hou1,2, Brent Pedersen1,2, Aaron Quinlan1,2,3
1Department of Human Genetics, University of Utah, Salt Lake City, UT, USA.
Nature Computational Science
|August 8, 2022
Summary
We developed the dense depth data dump (D4) format to improve quantitative genomics assay analysis. D4 balances faster analysis speeds and reduced file sizes, enabling scalable genomic data analysis.
Area of Science:
- Genomics
- Bioinformatics
- Computational Biology
Background:
- DNA sequencing is crucial for quantitative genomics assays, using read depth as a signal.
- Current data formats for quantitative genomics assays face limitations in analysis speed and file size.
- Efficient data handling is essential for scalable genomic data analysis.
Purpose of the Study:
- To develop a novel data format and tool suite for quantitative genomics assays.
- To address the limitations of existing formats regarding analysis speed and disk space.
- To enable faster and more scalable downstream genomic analyses.
Main Methods:
- Developed the dense depth data dump (D4) format and associated tool suite.
- Implemented an adaptive encoding strategy that profiles sequence depth for optimization.
- Benchmarked D4 against existing formats for random access, aggregation, and summarization.
Main Results:
- D4 format demonstrates substantial improvements in analysis speeds for random access, aggregation, and summarization.
- D4 achieves comparable or smaller file sizes compared to existing quantitative genomics data formats.
- The D4 format effectively balances analysis performance with file size efficiency.
Conclusions:
- The D4 format offers significant advantages for quantitative genomics data analysis.
- D4 facilitates scalable downstream analyses that were previously computationally challenging.
- This new format represents a significant advancement in handling quantitative genomics data.
