STDADS: An Efficient Slow Task Detection Algorithm for Deadline Schedulers

Area of Science:

• Computer Science
• Distributed Systems
• Big Data Analytics

Background:

• Apache Hadoop, an open-source implementation of Google's MapReduce, processes large datasets.
• Hadoop MapReduce and HDFS manage data and processing efficiently.
• Speculative task execution improves Quality of Service (QoS) by re-executing tasks on other nodes.

Purpose of the Study:

• To address the lack of integration between speculative task execution mechanisms and Deadline Schedulers in Hadoop.
• To present an improved speculative task detection algorithm tailored for Deadline Schedulers.

Main Methods:

• Developed a novel speculative task detection algorithm specifically for Hadoop Deadline Schedulers.
• Integrated node performance tracking for more efficient speculative task re-execution.
• Evaluated the algorithm's impact on QoS metrics for jobs with deadlines.

Main Results:

• Successfully improved QoS for Hadoop clusters handling deadline-driven jobs.
• Demonstrated enhancements in the percentage of successfully completed jobs.
• Reduced speculative task detection time and improved detection accuracy.
• Decreased the percentage of incorrectly flagged speculative tasks.

Conclusions:

• Regular tracking of node performance is crucial for efficient speculative task execution.
• The proposed algorithm significantly enhances Hadoop cluster performance for deadline-based workloads.
• The improved algorithm offers better QoS by optimizing speculative task handling in deadline-aware scheduling.