Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

Area of Science:

• Computer Science
• Distributed Systems
• Data Analytics

Background:

• Distributed computing, particularly cloud computing since 2006, has driven advancements in data collection and analysis models.
• Hadoop's MapReduce framework is central to processing large-scale data, with speculative execution addressing straggler issues.
• Accurate estimation of run-time task execution times remains a challenge, impacting efficient task allocation and distribution in MapReduce.

Purpose of the Study:

• To address the lack of efficient solutions for predicting MapReduce task execution times.
• To improve the accuracy of task completion time estimations for better resource management.
• To enhance the overall efficiency of large-scale data processing in distributed environments.

Main Methods:

• Collection and analysis of detailed task execution data within the Hadoop MapReduce framework.
• Development and application of a novel two-phase regression (TPR) method for time prediction.
• Evaluation of the TPR method's accuracy in estimating concurrent task execution times.

Main Results:

• The proposed two-phase regression method significantly improves the prediction accuracy of task finishing times.
• The method demonstrates particular effectiveness for regular job types within MapReduce.
• Detailed analysis of task execution data provides insights into performance characteristics.

Conclusions:

• The two-phase regression method offers an efficient solution for accurate task execution time estimation in MapReduce.
• Improved time prediction enhances task allocation and distribution, optimizing distributed computing performance.
• This approach contributes to more reliable and efficient large-scale data analytics.