Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques
Muntadher Saadoon1, Siti Hafizah Ab Hamid1, Hazrina Sofian1
1Department of Software Engineering, Faculty of Computer Science and Information Technology, University Malaya, Kuala Lumpur 50603, Malaysia.
Sensors (Basel, Switzerland)
|June 2, 2021
Summary
Hadoop MapReduce fault recovery causes significant response time penalties. Optimizing fault tolerance requires considering data locality and resource availability alongside detection and recovery times.
Area of Science:
- Computer Science
- Distributed Systems
- Fault Tolerance
Background:
- Hadoop MapReduce uses reactive fault detection and recovery, leading to high response times and resource inefficiency.
- Existing solutions neglect critical factors like fail-slow faults and infrastructure-level impacts.
Purpose of the Study:
- Analyze response time penalties under fail-stop and fail-slow fault conditions.
- Investigate the relationship between fault detection and recovery times.
- Evaluate fault tolerance in Hadoop MapReduce, YARN, and HDFS.
Main Methods:
- Experimental analysis on a real Hadoop cluster.
- Simulation of fail-stop and fail-slow faults at node, service, and task levels.
- Focus on the interplay between detection and recovery durations.
Main Results:
- Single fault recovery incurs an average response time penalty of 67.6%.
- Detection and recovery times, while optimized, do not guarantee optimal fault tolerance.
- Data locality and resource availability significantly influence fault tolerance.
Conclusions:
- Current Hadoop MapReduce fault tolerance mechanisms are inefficient.
- Achieving optimal fault tolerance requires a holistic approach considering detection, recovery, data locality, and resource availability.
- Further research is needed to mitigate response time penalties in distributed systems.
Related Concept Videos
01:18Fault Types
175
When analyzing a single line-to-ground fault from phase A to ground at a three-phase bus, it is important to consider the fault impedance. This impedance is zero for a bolted fault, equal to the arc impedance for an arcing fault, and represents the total fault impedance for a transmission-line insulator flashover. To derive sequence and phase currents, fault conditions are translated from the phase domain to the sequence domain.
For line-to-line faults occurring between phases B and C, the...
For line-to-line faults occurring between phases B and C, the...
175
01:12Types of Errors: Detection and Minimization
6.5K
Error is the deviation of the obtained result from the true, expected value or the estimated central value. Errors are expressed in absolute or relative terms.
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
6.5K
01:00Detection of Gross Error: The Q Test
6.6K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.6K
01:21Distributed Loads: Problem Solving
863
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
863
01:15Elastic Collisions: Case Study
15.8K
Elastic collision of a system demands conservation of both momentum and kinetic energy. To solve problems involving one-dimensional elastic collisions between two objects, the equations for conservation of momentum and conservation of internal kinetic energy can be used. For the two objects, the sum of momentum before the collision equals the total momentum after the collision. An elastic collision conserves internal kinetic energy, and so the sum of kinetic energies before the collision equals...
15.8K
01:11Hazard Rate
239
The hazard rate, also known as the hazard function or failure rate, is a statistical measure used to describe the instantaneous rate at which an event occurs, given that the event has not yet happened. From a probabilistic perspective, it represents the likelihood that a subject will experience the event in a very small time interval, conditional on surviving up to the beginning of that interval. In terms of frequency, the hazard rate can be viewed as the ratio of the number of events to the...
239