WG-Storm:用于分布式流处理引擎的资源意识调度器
Rizwan Ali1, Asif Muhammad1, Muhammad Aleem2
1Department of Software Engineering, National University of Computer and Emerging Sciences, Islamabad, Islamabad, Punjab, Pakistan.
PeerJ. Computer science
|June 26, 2025
概括
本研究介绍了WG-Storm,这是一款用于流处理引擎 (SPE) 的新型调度器. 通过考虑拓和资源意识,WG-Storm可以提高大数据应用程序的资源利用率和吞吐量.
科学领域:
- 计算机科学 计算机科学
- 分布式系统 分布式系统
- 大数据分析大数据分析
背景情况:
- 流处理引擎 (SPEs) 面临着由于资源利用,动态配置和异质环境而导致大数据应用程序调度的挑战.
- 越来越多的数据量使资源和应用程序需求的预测变得复杂,影响了整体系统吞吐量.
- 现有的SPEs经常忽视网络拓,导致吞吐量最小化和延迟增加.
研究的目的:
- 提出一个拓意识和资源意识的调度器,WG-Storm,以提高资源使用率和流处理中的吞吐量.
- 解决任务分配中的低效率问题,这些问题限制了大数据应用程序的最大吞吐量.
- 为了提高Apache Storm在异质集群环境中的性能.
主要方法:
- 开发了WG-Storm,这是一个基于指向非循环图 (DAG) 的调度器,集成了拓和资源意识.
- 在Apache Storm平台上实现了WG-Storm.
- 评估了WG-Storm使用两个线性拓,并将其性能与五个最先进的调度器进行了比较.
主要成果:
- 与现有调度器相比,WG-Storm的吞吐量增加了多达30%.
- 拟议的调度器实现了更高的吞吐量,同时使用更少的计算资源.
- 实验结果证实了在异质集群中改善了资源使用和整体系统效率.
结论:
- WG-Storm有效地提高了流处理引擎的资源利用率和整体吞吐量.
- 拓意识和资源意识的方法显著提高了任务分配效率.
- 在复杂的环境中,WG-Storm为优化大数据应用程序调度提供了一个有希望的解决方案.
相关概念视频
Parallel Processing
252
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
252
Rapidly Varying Flow
146
Rapidly varying flow (RVF) in open channels is characterized by abrupt changes in flow depth over a short distance, with the rate of depth change relative to distance often approaching unity. These flows are inherently complex due to their transient and multi-dimensional nature, making exact analysis difficult. However, approximate solutions using simplified models provide valuable insights into their behavior.Key Features of Rapidly Varying FlowRVF is commonly observed in scenarios involving...
146
Distributed Loads
629
Distributed loads are a common type of load that engineers and scientists encounter in various practical situations. Distributed loads often refer to a type of load spread over a surface or a structure and can be modeled as continuous force per unit area.
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
629
Stream Function
1.6K
In two-dimensional incompressible fluid flow, the continuity equation is essential for ensuring mass conservation, meaning that any change in fluid entering or exiting a region is balanced by a corresponding change elsewhere. For incompressible flow, where density remains constant, this requirement simplifies to the condition that the divergence of the velocity field must be zero. Mathematically, this is expressed as,
1.6K
Precipitate Formation and Particle Size Control
992
In precipitation gravimetry, the precipitating agent should react specifically or selectively with the analyte. While a specific reagent reacts with the analyte alone, a selective reagent can react with a limited number of chemical species.
The obtained precipitate should be either a pure substance of known composition or easily converted to one by a simple process, such as ignition or drying. In addition, the precipitate should be insoluble and easily filterable. In general, filterability...
The obtained precipitate should be either a pure substance of known composition or easily converted to one by a simple process, such as ignition or drying. In addition, the precipitate should be insoluble and easily filterable. In general, filterability...
992
Distributed Loads: Problem Solving
744
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
744


