SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines

Area of Science:

• Bioinformatics
• Cheminformatics
• Computational Biology
• Machine Learning

Background:

• Biological data complexity necessitates specialized software tools.
• Scientific workflow management systems aid pipeline assembly, automation, and reproducibility.
• Existing workflow tools often lack support for complex constructs like nested loops and dynamic scheduling common in machine learning.

Purpose of the Study:

• Introduce SciPipe, a workflow programming library designed for complex and dynamic scientific pipelines.
• Address limitations of contemporary workflow tools, especially for machine learning tasks.
• Facilitate agile development and enhance reproducibility in scientific computing.

Main Methods:

• SciPipe is implemented in the Go programming language, utilizing flow-based programming principles.
• It supports complex workflow constructs including branching, parameter sweeps, and dynamic scheduling.
• Features include reusable components, subset execution for iterative development, and data-centric audit logging.

Main Results:

• SciPipe enables agile development of complex, dynamic pipelines for bioinformatics, cheminformatics, and machine learning.
• Demonstrated utility through machine learning, genomics, and transcriptomics pipelines.
• Provides a flexible API suitable for scientists familiar with programming.

Conclusions:

• SciPipe offers a robust solution for managing intricate and adaptive scientific workflows.
• It particularly benefits machine learning applications requiring advanced pipeline constructs.
• The library promotes efficient, reproducible, and agile scientific research.