Enhancing performance of E-Government information systems with SSD-based Hadoop mapreduce

Area of Science:

• Computer Science
• Data Engineering
• E-Government

Background:

• E-government systems generate vast, heterogeneous datasets requiring high-throughput, low-latency processing.
• Hadoop MapReduce, while common, faces performance bottlenecks due to disk I/O and network latency in the shuffle phase.

Purpose of the Study:

• To propose and evaluate a data address-based shuffle mechanism for Hadoop clusters utilizing Solid-State Drives (SSDs).
• To enhance data processing performance and efficiency for large-scale e-government applications.

Main Methods:

• Implementation of a novel shuffle mechanism featuring address-based sorting, merging, and pre-transmission of intermediate data.
• Performance evaluation using Terasort and Wordcount benchmarks on Hadoop clusters equipped with SSDs.
• Scalability testing on a simulated 50-node cluster and energy consumption profiling.

Main Results:

• Achieved execution time reductions of 8% (Terasort) and 1% (Wordcount) with statistically significant 95% confidence intervals.
• Demonstrated improved performance, reduced network congestion, and a 31% decrease in energy consumption compared to Hard Disk Drive (HDD) systems.
• Validated enhanced efficiency and scalability for large-scale data processing.

Conclusions:

• The proposed data address-based shuffle mechanism offers a cost-effective and efficient solution for big data processing in public sector computing.
• Optimizing shuffle mechanisms for SSDs is crucial for advancing e-government data processing capabilities.
• The approach effectively mitigates I/O and network overhead, leading to substantial performance gains.