Standardizing the next generation of bioinformatics software development with BioHDF (HDF5)

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Next Generation Sequencing (NGS) technologies generate vast amounts of data, necessitating efficient bioinformatics infrastructures.
• Existing data storage and processing methods for NGS data present limitations in scalability and integration.
• Standardization is lacking in bioinformatics infrastructures, hindering data management and analysis performance.

Purpose of the Study:

• To evaluate Hierarchical Data Format (HDF) technologies as a solution for Next Generation Sequencing bioinformatics challenges.
• To highlight the advantages of HDF5 and BioHDF for scalable data storage, processing, and integration in genomics.
• To propose HDF5-based systems as a suitable data model for next-generation bioinformatics applications.

Main Methods:

• Review of Hierarchical Data Format (HDF) technologies and their applications in data-intensive scientific communities.
• Comparative analysis of HDF5-based systems against emerging standards like SAM/BAM formats.
• Assessment of HDF5 features including scalability, multiple indexes, diverse data type storage, and self-describing capabilities.

Main Results:

• HDF technologies provide robust solutions for reducing data storage and increasing data processing performance.
• HDF5-based systems demonstrate superior scalability and support for multiple indexes compared to SAM/BAM formats.
• HDF5 and its BioHDF extension are capable of storing multiple data types and are self-describing.

Conclusions:

• HDF5 and BioHDF offer significant advantages for managing and analyzing large-scale Next Generation Sequencing data.
• These HDF technologies address key limitations in current bioinformatics infrastructures.
• HDF5-based data models are well-suited for supporting the evolving needs of next-generation bioinformatics applications.