MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase...

Area of Science:

• Computer Science
• Distributed Systems
• Algorithm Design

Background:

• Stream Processing Engines (SPEs) manage computational tasks on clusters, but optimal resource mapping remains a challenge.
• Inefficient scheduling leads to increased network latency, reduced resource utilization, and lower cluster throughput.
• Existing methods struggle to balance computational and communication demands of streaming applications.

Purpose of the Study:

• To address the gap in optimal task scheduling for streaming applications on heterogeneous clusters.
• To develop a scheduler that maximizes throughput and resource utilization.
• To minimize scheduling cost while considering dynamic resource availability.

Main Methods:

• Introduced MF-Storm, a max-flow min-cut based job scheduler.
• MF-Storm partitions application task graphs to minimize inter-partition traffic.
• Assigns partitions to computing nodes based on their computational power for efficient execution.

Main Results:

• MF-Storm demonstrated an average throughput improvement of 148%.
• Achieved this improvement using 30% fewer computational resources compared to state-of-the-art schedulers.
• Experimental validation on a physical cluster confirmed significant performance gains.

Conclusions:

• MF-Storm effectively optimizes scheduling for streaming applications.
• The max-flow min-cut approach significantly enhances throughput and resource efficiency.
• This scheduler offers a near-optimum solution for complex computational cluster environments.