Exploring the effects and potential of unlocked I/O-powered single board computer clusters

Area of Science:

• Computer Science
• Data Engineering
• High-Performance Computing

Background:

• Big data processing relies on Hadoop and Spark, driving demand for powerful servers, leading to high costs and energy consumption.
• Single board computer (SBC) clusters present an energy-efficient alternative, but limitations in storage I/O hindered big data processing.
• The Raspberry Pi 5B's introduction of a PCIe interface enables high-performance storage, overcoming previous SBC limitations.

Purpose of the Study:

• To investigate the feasibility and potential of SBC clusters for terabyte-scale big data processing.
• To evaluate the performance of a Hadoop cluster built with Raspberry Pi 5B SBCs and PCIe-based SSDs.
• To identify and discuss challenges and optimization strategies for SBC-based big data processing.

Main Methods:

• Constructed a Hadoop cluster using Raspberry Pi 5B (8 GB RAM) single board computers.
• Integrated high-speed, PCIe-based Solid State Drives (SSDs) for enhanced storage performance.
• Conducted six established big data benchmarks using datasets up to 2 TB.

Main Results:

• The Raspberry Pi 5B cluster with PCIe SSDs demonstrated significant potential for terabyte-scale big data processing.
• Performance was influenced by factors including I/O throughput, CPU overclocking, power supply, and SSD trim commands.
• Optimized SBC configurations can achieve practical big data processing capabilities.

Conclusions:

• Raspberry Pi 5B SBC clusters with PCIe-enabled SSDs offer a cost-effective and energy-efficient solution for big data challenges.
• Addressing I/O bottlenecks and optimizing system parameters are crucial for maximizing SBC cluster performance.
• This approach paves the way for practical, real-world big data processing using SBC clusters.