metEAUdata: a framework for automatic metadata generation for environmental time series preprocessing

Area of Science:

• Environmental Engineering
• Data Science
• Wastewater Treatment

Background:

• Water resource recovery facilities generate vast amounts of time series data from supervisory control and data acquisition (SCADA) systems and process historians.
• Essential data pre-processing steps like artifact removal, fault detection, smoothing, and aggregation often discard valuable metadata, compromising data context, traceability, and reusability.
• Manual metadata reporting is difficult and time-consuming, especially given the iterative nature of data pre-processing.

Purpose of the Study:

• To present metEAUdata, a novel framework designed to automate the creation and preservation of metadata during data pre-processing operations.
• To enable seamless integration of metadata awareness into existing scientific libraries and pre-processing algorithms with minimal effort.
• To facilitate the transition towards FAIR (Findable, Accessible, Interoperable, Reusable)-compliant time series data in the water sector.

Main Methods:

• Development of metEAUdata, a lightweight framework with a simple interface for metadata-aware algorithm integration.
• Implementation of wrapper functions to enable arbitrary algorithms to automatically generate and record metadata during pre-processing.
• Focus on preserving both original metadata and descriptions of the pre-processing steps themselves.

Main Results:

• metEAUdata successfully automates the creation of structured and extensive metadata for diverse pre-processing pipelines.
• The framework allows existing scientific libraries to become metadata-aware with minimal code modification.
• Demonstrated ability to maintain data context, traceability, and reusability throughout the pre-processing workflow.

Conclusions:

• metEAUdata addresses the critical challenge of metadata loss in water resource recovery facility data pre-processing.
• The framework promotes FAIR data principles by ensuring comprehensive metadata capture and preservation.
• Automated metadata management via metEAUdata enhances the value and usability of time series data for analysis and decision-making.