polars-bio-fast, scalable, and out-of-core operations on large genomic interval datasets

Area of Science:

• Computational Genomics
• Bioinformatics
• Data Science

Background:

• Genomic studies require computationally intensive analyses of genomic intervals.
• Python, with libraries like Pandas, is widely used but faces scalability challenges.
• Polars, a modern alternative, offers improved performance with a Rust backend.

Purpose of the Study:

• Introduce polars-bio, a Python library for efficient genomic interval data processing.
• Enable fast, parallel, and out-of-core operations on large genomic datasets.
• Provide a compatible alternative to existing libraries like Bioframe.

Main Methods:

• Implemented in Rust using Apache DataFusion and Apache Arrow.
• Compatible with Polars and Pandas DataFrame formats.
• Benchmarked against Bioframe on real-world and synthetic datasets.

Main Results:

• Achieved significant speedups: 6.5x-38x faster than Bioframe on real-world data.
• Demonstrated substantial memory reductions: up to 90x less memory usage.
• Showcased good scalability characteristics in multi-threaded benchmarks.

Conclusions:

• polars-bio is the most efficient single-node Python library for genomic interval DataFrames.
• Offers substantial performance improvements for genomic data analysis.
• Facilitates more efficient handling of large-scale genomic interval datasets.