BEETL-fastq: a searchable compressed archive for DNA reads

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• FASTQ is a standard format for DNA sequencing data, generating terabytes of information.
• Existing compression tools like gzip require decompression before data use.
• Efficient storage and retrieval of large sequencing datasets are critical challenges.

Purpose of the Study:

• To introduce BEETL-fastq, a novel tool for compressing and searching DNA sequencing data.
• To offer a compression method superior to gzip for FASTQ files.
• To enable rapid k-mer searching within compressed FASTQ archives.

Main Methods:

• BEETL-fastq compresses FASTQ files, reducing storage requirements.
• The tool creates indexed files for efficient data retrieval.
• It allows direct searching for k-mers within the compressed data.

Main Results:

• BEETL-fastq compresses 6.6 TB of human reads to 1.7 TB.
• Searching for a million 30-mers takes only 567 seconds.
• The tool returns full FASTQ records for matching reads, enabling downstream analysis.

Conclusions:

• BEETL-fastq provides significant data compression for FASTQ files.
• It enables rapid and efficient searching of large-scale sequencing data.
• The tool supports direct integration with existing bioinformatics workflows for variant calling and visualization.