Automated Execution of Data Pipelines based on Configuration Files

Area of Science:

• Data Engineering
• Software Frameworks
• Computational Science

Background:

• Data preparation is crucial for data visualization, reporting, and machine learning model training.
• Data transformation processes often lack standardization due to varying project contexts, data sources, and application domains.

Purpose of the Study:

• To present a flexible and extensible framework for coordinated execution of modular data processing steps.
• To improve code reuse, maintainability, and adaptability in data engineering workflows through a declarative, configuration-driven approach.

Main Methods:

• Developed a framework enabling coordinated execution of modular data processing steps defined in a configuration file.
• Implemented a declarative, configuration-driven approach to data pipeline development.
• Incorporated iterative execution constructs (loops, recursion) for complex workflows.

Main Results:

• The framework promotes modular, step-by-step development, enhancing code reuse and maintainability.
• Facilitates the creation of a shared codebase, reducing project-specific implementations.
• Demonstrated reduced development effort in simple data engineering scenarios using cyclomatic complexity.

Conclusions:

• The proposed framework enhances flexibility, maintainability, and adaptability in data engineering.
• Shifts development focus from disposable implementations to refining a shared, reusable code base.
• Offers long-term benefits through generalized modules, reducing duplication and improving efficiency.