Pithos - a scalable and secure data container for FAIR-compliant research data management in life sciences

Area of Science:

• Life Sciences
• Bioinformatics
• Computational Biology

Background:

• Scientific data is growing exponentially, posing challenges for secure and efficient management.
• Current file formats and data storage methods in life sciences lead to fragmented data, reproducibility issues, and security/accessibility trade-offs.
• Large-scale genomic studies face bottlenecks due to large file sizes and inefficient cloud data access.

Purpose of the Study:

• Introduce Pithos, a novel file format designed for scientific data management in distributed cloud environments.
• Address limitations of traditional file formats in handling large, complex scientific datasets.
• Enhance data security, accessibility, and research reproducibility.

Main Methods:

• Developed Pithos, a next-generation file format utilizing content-defined chunking.
• Implemented an append-only structure for data immutability and incremental updates.
• Conducted benchmark tests comparing Pithos performance against existing solutions.

Main Results:

• Pithos enables efficient deduplication across distributed storage, reducing costs and bandwidth.
• The format ensures data immutability and supports incremental updates.
• Benchmark results demonstrate superior read/write performance and comparable/improved storage efficiency compared to existing formats.

Conclusions:

• Pithos offers a significant advancement in scientific data management for cloud environments.
• The format effectively balances security, accessibility, and performance for large-scale scientific data.
• Pithos has the potential to improve research reproducibility and reduce computational bottlenecks in life sciences.