Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit.

Area of Science:

• Bioinformatics
• Computational Biology
• Scientific Data Management

Background:

• Life science applications generate large, resource-intensive data streams.
• Processing, transporting, and storing these datasets pose significant challenges.
• A novel pipeline model is proposed to manage these data streams effectively.

Purpose of the Study:

• To introduce a pipeline model for organizing scientific data streams into a tiered data hierarchy.
• To present the HASTE Toolkit, a cloud-native software toolkit implementing this model.
• To optimize the use of limited computing resources for data-intensive life science applications.

Main Methods:

• An "interestingness function" scores data objects, creating a data hierarchy.
• A "policy" guides computational resource allocation based on data object scores.
• The HASTE Toolkit provides tools to implement this prioritization strategy.
• Evaluation involved two microscopy imaging case studies: high content screening and transmission electron microscopy edge processing.

Main Results:

• Developed smart data pipelines that efficiently utilize storage, compute, and network resources.
• Demonstrated effective prioritization of data streams in both on-premise and cloud environments.
• Achieved more efficient data-intensive experiments through optimized resource allocation.

Conclusions:

• The pipeline model and HASTE Toolkit enable efficient resource management for large-scale scientific data.
• A clear separation between data prioritization logic and resource implementation is beneficial.
• The toolkit offers a flexible solution for intelligent data prioritization in diverse deployment scenarios.