An Efficient Group-Based Replica Placement Policy for Large-Scale Geospatial 3D Raster Data on Hadoop

Area of Science:

• Geographic Information Science
• Computer Science
• Data Engineering

Background:

• Geospatial 3D raster data are crucial for various analyses but generate massive datasets.
• Current Hadoop processing of this data ignores spatial characteristics, causing network bottlenecks.
• Exponential growth in data resolution and accuracy exacerbates performance issues.

Purpose of the Study:

• To propose an efficient group-based replica placement policy for large-scale geospatial 3D raster data.
• To optimize replica locations within Hadoop clusters to minimize network overhead.
• To maintain Hadoop's replica placement requirements while improving data processing efficiency.

Main Methods:

• Developed an overlapped group scheme for placing three replicas of each file.
• Ensured data within each group resides on the same datanode.
• Implemented diverse colocation patterns for replicas to reduce inter-group communication.

Main Results:

• Demonstrated significant reduction in network overhead for 3D raster data acquisition in Hadoop clusters.
• Validated the effectiveness of the group-based replica placement strategy.
• Confirmed that the proposed method adheres to Hadoop's replica placement requirements.

Conclusions:

• The proposed group-based replica placement policy effectively addresses network overhead issues in processing large-scale geospatial 3D raster data.
• Optimizing replica placement is key to enhancing the performance of big data analytics for spatial datasets.
• This approach offers a practical solution for managing and processing massive geospatial data in distributed environments.