NApy: efficient statistics in Python for large-scale heterogeneous data with enhanced support for missing data.

Area of Science:

• Computational Biology
• Data Science
• Statistical Computing

Background:

• Existing Python libraries struggle with efficient statistical testing on large datasets containing missing values.
• Runtime and memory constraints are critical for applications like interactive biomedical data analysis.
• This limitation hinders exploratory data analysis in resource-intensive fields.

Purpose of the Study:

• To introduce NApy, a Python package designed for scalable statistical testing.
• To address the challenge of handling missing values in large, mixed-type datasets.
• To provide an efficient solution for computational tasks in data science and bioinformatics.

Main Methods:

• Developed NApy with a Numba and C++ backend.
• Implemented OpenMP for parallelization to enhance performance.
• Focused on optimizing statistical test computations for datasets with missing entries.

Main Results:

• NApy demonstrates significant improvements in runtime and memory consumption.
• Outperforms existing tools and naive parallelization methods by orders of magnitude.
• Enables efficient on-the-fly statistical analyses for interactive applications.

Conclusions:

• NApy offers a scalable and efficient solution for statistical testing with missing data.
• The package facilitates real-time data analysis in interactive environments.
• NApy is publicly available, promoting its adoption in research and industry.