Analysis-ready VCF at Biobank scale using Zarr.

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Variant Call Format (VCF) is the standard for genetic variation data but is inefficient for large-scale biobanks.
• Row-wise encoding of VCF is unsuitable for hundreds of terabytes of genomic data.
• A more scalable approach is needed to handle increasing dataset sizes.

Purpose of the Study:

• To introduce the VCF Zarr specification for efficient storage and processing of genetic variation data.
• To provide software infrastructure for large-scale conversion to the VCF Zarr format.
• To demonstrate the performance benefits of VCF Zarr over traditional VCF approaches.

Main Methods:

• Encoding the VCF data model using the Zarr format for multidimensional data storage.
• Developing software for efficient and reliable large-scale data conversion.
• Benchmarking VCF Zarr against standard VCF and specialized methods using large human and non-human genomic datasets.

Main Results:

• VCF Zarr demonstrates significantly higher efficiency compared to standard VCF.
• Compression ratios and single-threaded performance are competitive with specialized genotype storage methods.
• Case studies on large human datasets (Genomics England, Our Future Health, All of Us) and whole-genome datasets (Norway Spruce, SARS-CoV-2) show promising results.
• Illustrative examples highlight the potential for cloud computing and GPU acceleration with VCF Zarr.

Conclusions:

• Large row-encoded VCF files present a significant bottleneck and cost in current research.
• The VCF Zarr specification, based on open-source technologies, can substantially reduce storage and processing costs.
• VCF Zarr has the potential to foster a new ecosystem of cloud-native tools for genetic variation analysis while maintaining compatibility with existing workflows.