Scalable high-performance single cell data analysis with BPCells.

Area of Science:

• Computational Biology
• Genomics
• Bioinformatics

Background:

• Single-cell analysis software faces scalability challenges with multi-million cell datasets.
• Existing workflows often require substantial memory, limiting accessibility.

Purpose of the Study:

• To introduce BPCells, a package for high-performance single-cell analysis.
• To address memory limitations in analyzing large-scale RNA-seq and ATAC-seq data.

Main Methods:

• Utilizes disk-backed streaming compute algorithms to minimize memory footprint.
• Implements novel, high-performance compressed formats using bitpacking for ATAC-seq fragment files and sparse matrices.
• Evaluates compression algorithms for computational overhead and data transfer efficiency.

Main Results:

• Achieves memory requirement reductions of nearly 70-fold compared to in-memory methods.
• Maintains execution speed with minimal performance loss.
• Successfully performs normalization and PCA on a 44 million cell dataset using a laptop.
• Demonstrates feasibility of analyzing large single-cell datasets on modest hardware.

Conclusions:

• BPCells enables analysis of contemporary large-scale single-cell datasets on accessible hardware.
• The package provides efficient memory management and accelerated disk-backed analysis.
• BPCells offers a scalable solution for current and future single-cell data analysis needs.