BINSEQ: A family of high-performance binary formats for nucleotide sequences

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Modern genomics generates massive sequencing data, typically stored in gzip-compressed FASTQ files.
• FASTQ format's single-threaded decompression and sequential parsing limit high-throughput parallel processing.
• This bottleneck affects critical applications like read mapping, variant calling, and de novo assembly.

Purpose of the Study:

• Introduce BINSEQ, a family of binary formats designed for high-throughput parallel processing of sequencing data.
• Optimize storage and access for large-scale genomics analyses.
• Facilitate adoption through accessible libraries and tools.

Main Methods:

• Developed BINSEQ family with two implementations: BQ (fixed-length reads, 2/4-bit encoding, random access) and VBQ (variable-length sequences, optional quality scores, block compression).
• Implemented high-performance libraries for reading/writing BINSEQ.
• Provided native parallelization strategies and a command-line conversion tool.

Main Results:

• BINSEQ files achieve up to 90x speedup compared to compressed FASTQ for parallel processing.
• Demonstrated significant reduction in analysis time from hours to minutes for large-scale genome and transcriptome analyses.
• Validated efficiency for resource-intensive applications such as alignment, mapping, and de novo assembly.

Conclusions:

• BINSEQ formats offer a substantial improvement for high-throughput parallel processing of sequencing data.
• Enable faster and more efficient large-scale genomics analyses.
• Facilitate broader adoption of efficient data processing in genomics research.