基于配置文件的数据管道自动执行
Károly Bósa1, Paul Heinzlreiter1
1RISC Software GmbH, Hagenberg, 4232, Austria.
Open research Europe
|January 12, 2026
概括
本研究介绍了用于模块化数据处理的灵活数据工程框架. 配置驱动的方法提高了复杂数据工作流程的代码重复使用和维护能力.
科学领域:
- 数据工程数据工程
- 软件框架 软件框架
- 计算科学 计算科学
背景情况:
- 数据准备对于数据可视化,报告和机器学习模型培训至关重要.
- 数据转换过程往往缺乏标准化,因为项目背景,数据源和应用领域各不相同.
研究的目的:
- 为协调执行模块化数据处理步骤提供灵活和可扩展的框架.
- 通过声明式,配置驱动的方法,提高代码的重复使用,可维护性和数据工程工作流程的适应性.
主要方法:
- 开发了一个框架,允许在配置文件中定义的模块化数据处理步骤的协调执行.
- 实施了声明式的,以配置为导向的数据管道开发方法.
- 集成的代执行构造 (循环,递归) 用于复杂的工作流.
主要成果:
- 该框架促进模块化,逐步开发,增强代码的重复使用和可维护性.
- 方便创建共享代码库,减少项目特定的实施.
- 在使用周期复杂性的简单数据工程场景中,证明了开发工作的减少.
结论:
- 拟议的框架提高了数据工程的灵活性,可维护性和适应性.
- 开发重点从一次性实现转移到改进一个共享的,可重复使用的代码库.
- 通过通用模块提供长期好处,减少重复和提高效率.
更多相关视频
09:43Databases to Efficiently Manage Medium Sized, Low Velocity, Multidimensional Data in Tissue Engineering
Published on: November 22, 2019
6.7K
10:44Inherent Dynamics Visualizer, an Interactive Application for Evaluating and Visualizing Outputs from a Gene Regulatory Network Inference Pipeline
Published on: December 7, 2021
2.6K
相关概念视频
01:25Distribution Reliability and Automation
488
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
488
01:21Multiple Pipe Systems
1.2K
Multipipe systems consist of complex configurations of interconnected pipes designed to transport fluids efficiently across intricate networks. They are essential in engineering applications requiring precise control over flow distribution, pressure, and head loss. They are categorized into series, parallel, loop, and network configurations, each distinguished by unique flow characteristics and applications.
Series Configuration
In a series configuration, fluid flows sequentially from one pipe...
Series Configuration
In a series configuration, fluid flows sequentially from one pipe...
1.2K