HaRD: a heterogeneity-aware replica deletion for HDFS

Area of Science:

• Distributed Systems
• Big Data Storage
• Data Management

Background:

• Hadoop Distributed File System (HDFS) uses data replication for reliability and throughput, balancing availability against disk usage.
• Dynamic replication schemes adjust file replication factors based on data popularity to optimize performance.
• Reducing replication factors for less popular data can lead to unbalanced distribution, causing performance issues like hot spots and increased network usage.

Purpose of the Study:

• To confirm that reducing replication factors in HDFS leads to unbalanced data distribution.
• To demonstrate the sub-optimal performance of existing balanced distribution schemes (WBRD) on heterogeneous clusters.
• To propose and evaluate a new heterogeneity-aware replica deletion scheme (HaRD) for HDFS.

Main Methods:

• Validated the impact of replication factor reduction on data distribution using Hadoop's default scheme.
• Implemented the proposed heterogeneity-aware replica deletion scheme (HaRD) on HDFS.
• Conducted performance evaluations on a 23-node dedicated heterogeneous cluster.

Main Results:

• Confirmed that dynamic replication factor reduction causes unbalanced data distribution.
• Showed that WBRD, while balancing distribution, is sub-optimal on heterogeneous clusters.
• HaRD reduced execution time by up to 60% compared to default Hadoop and 17% compared to WBRD.

Conclusions:

• The proposed HaRD scheme effectively addresses data distribution imbalance in heterogeneous HDFS clusters.
• HaRD leverages node processing capabilities to optimize replica placement, enhancing overall system performance.
• HaRD offers significant improvements in execution time for large-scale data processing on diverse hardware.