Experimental Directory Structure (Exdir): An Alternative to HDF5 Without Introducing a New File Format.

Area of Science:

• Scientific data management
• Open data formats
• Computational neuroscience

Background:

• Natural sciences generate vast amounts of diverse data, necessitating effective data sharing for reproducible research.
• Hierarchical Data Format 5 (HDF5) is a widely adopted open format but has limitations including complexity and lack of human-readability.
• Existing data organization methods lack a common standard, hindering data sharing and tool development.

Purpose of the Study:

• To propose Experimental Directory Structure (Exdir) as a novel, open specification for experimental data storage.
• To address the drawbacks of HDF5 by offering a more accessible and version-control-friendly alternative.
• To foster a collaborative development environment for a universal data storage standard.

Main Methods:

• Exdir utilizes a file system directory structure to represent data hierarchy.
• Metadata is stored in human-readable YAML files, and datasets in binary NumPy files.
• Raw data is stored directly in subdirectories, enhancing accessibility and version control compatibility.

Main Results:

• Exdir offers advantages over HDF5, including human-readability, improved version control integration, and easier external application access.
• The Exdir specification is compatible with the HDF5 Abstract Data Model, facilitating transition.
• Exdir promotes improved data storage, sharing, and interdisciplinary research.

Conclusions:

• Exdir provides a flexible and accessible open specification for experimental data storage.
• Adoption of Exdir can enhance data sharing, reproducibility, and collaborative research.
• The scientific community is invited to contribute to the ongoing development of the Exdir standard.