MRS-DP: Improving Performance and Resource Utilization of Big Data Applications with Deadlines and Priorities

Area of Science:

• Computer Science
• Data Science
• Distributed Computing

Background:

• Big data analytics is revolutionized by large datasets characterized by volume, velocity, variety, veracity, valence, and value.
• Timely data insights are crucial for decision-making, as delays can lead to missed opportunities.

Purpose of the Study:

• To introduce a novel MapReduce scheduler with deadline and priorities (MRS-DP) for efficient big data processing.
• To improve resource utilization, job progress monitoring, and backup mechanisms in distributed systems.

Main Methods:

• Development of the MRS-DP scheduler to handle jobs with deadlines and priorities.
• Evaluation through multiple workloads including WordCount and DataSort jobs.
• Comparative analysis against minimal earliest deadline first-work conserving scheduler and MapReduce Constraint Programming based Resource Management algorithm.

Main Results:

• The MRS-DP scheduler demonstrated a 10%-20% improvement in the percentage of successful jobs.
• Effective resource utilization increased by 20%-25%.
• The scheduler successfully ensured the offered quality of service (QoS) and avoided starvation of lower priority jobs.

Conclusions:

• The proposed MRS-DP scheduler offers significant improvements in big data processing efficiency and resource management.
• MRS-DP effectively balances job priorities and deadlines, maximizing throughput and QoS.
• This scheduler is a valuable contribution to optimizing distributed data analytics.