The scalable variant call representation: enabling genetic analysis beyond one million genomes

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• The Variant Call Format (VCF) is a standard for storing genomic variation data.
• VCF exhibits poor scalability with increasing sample sizes, leading to excessively large files (e.g., 900 TiB for 150,000 genomes).
• This inefficiency stems from dense representation of reference genotypes and allele-indexed arrays, causing data duplication.

Purpose of the Study:

• To introduce a novel data representation, the Scalable Variant Call Representation (SVCR), designed for efficient handling of large-scale genomic datasets.
• To demonstrate SVCR's ability to achieve linear scalability with the number of samples.
• To provide practical implementations and tools that facilitate the generation, analysis, and dissemination of massive genetic datasets.

Main Methods:

• Developed the Scalable Variant Call Representation (SVCR) incorporating local allele indices and reference blocks.
• Implemented SVCR in two formats: SVCR-VCF (VCF-compatible) and VDS (Hail's native format).
• Introduced the VDS Combiner tool for scalable VDS production from GVCFs.

Main Results:

• SVCR achieves linear file size scaling with the number of samples, unlike the super-linear growth of standard VCF.
• SVCR is lossless and supports incremental joint-calling (N+1 and N+K).
• Experimental results confirm the linear scalability of SVCR-VCF and VDS implementations.

Conclusions:

• SVCR effectively addresses the scalability limitations of VCF for large-scale genome sequencing projects.
• The VDS format and associated tools enable efficient generation and analysis of massive genetic datasets.
• SVCR and VDS empower the scientific community to handle and share genomic data from millions of samples.