MRS-DP: Improving Performance and Resource Utilization of Big Data Applications with Deadlines and Priorities

This article proposes the MapReduce scheduler with deadline and priorities (MRS-DP) scheduler capable of handling jobs with deadlines and priorities. Big data have emerged as a key concept and revolutionized data analytics in the present era. Big data are characterized by multiple dimensions or Vs, namely volume, velocity, variety, veracity, and valence. Recently, a new and important dimension (another V) is added, known as value. Value has emerged as an important characteristic and it can be understood in terms of delay in acquiring information, leading to late decisions that may result in missed opportunities. To gain optimal benefits, this article introduces a scheduler based on jobs with deadlines and priorities intending to improve resource utilization, with efficient job progress monitoring and backup launching mechanism. The proposed scheduler is capable of accommodating multiple jobs to maximize the number of jobs processed successfully and avoid starvation of lower priority jobs while improving the resource utilization and ensuring the assured quality of service (QoS). To evaluate our proposed scheduler, we ran multiple workloads consisting of the WordCount jobs and DataSort jobs. The performance of the proposed MRS-DP scheduler is compared with minimal earliest deadline first-work conserving scheduler and MapReduce Constraint Programming based Resource Management algorithm in terms of the percentage of successful jobs, priority-wise jobs, and resource utilization of the cluster. The result of the proposed scheduler depicts an improvement of around 10%-20% in terms of the percentage of successful jobs, 20%-25% concerning effective resource utilization offered, and the ability to ensure the offered QoS.