AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
Sebastiaan P Huber1,2, Spyros Zoupanos3,4, Martin Uhrin3,4
1National Centre for Computational Design and Discovery of Novel Materials (MARVEL), École Polytechnique Fédérale de Lausanne, CH-1015, Lausanne, Switzerland. mail@sphuber.net.
Scientific Data
|September 9, 2020
Summary
AiiDA (aiida.net) is an open-source infrastructure for managing scientific workflows and data provenance. It supports high-throughput computing and enables robust, reproducible simulations through automated data handling and analysis.
Area of Science:
- Computational science
- Scientific computing
- Data management
Background:
- Increasing computational power and advanced methods drive scientific progress.
- Managing large-scale calculations and data presents significant challenges.
- Exascale computing necessitates automated and scalable solutions for workflow and data management.
Purpose of the Study:
- Introduce developments and capabilities of the AiiDA infrastructure.
- Address challenges in automated workflow management and data provenance.
- Enable high-throughput scientific computing and data analytics.
Main Methods:
- Developed AiiDA, an open-source high-throughput infrastructure.
- Implemented automated workflow management and data provenance recording.
- Utilized a relational database for queryable and traversable data storage.
Main Results:
- AiiDA supports tens of thousands of processes per hour.
- Full data provenance is automatically preserved and stored.
- Enables high-performance data analytics through queryable data.
Conclusions:
- AiiDA provides robust, scalable solutions for computational challenges.
- Advanced automation and error handling enhance simulation robustness.
- A plugin model and registry foster community-driven development for reproducible research.
