Toward Millions of File System IOPS on Low-Cost, Commodity Hardware

Area of Science:

• Computer Science
• Data Storage Systems
• Operating Systems

Background:

• Traditional storage systems face Input/Output (I/O) bottlenecks, limiting performance, especially with Solid State Drives (SSDs).
• Non-uniform Memory Access (NUMA) architectures introduce complexities in memory and I/O scheduling, potentially causing CPU overhead and contention.
• Existing page caching mechanisms in operating systems may not scale efficiently for extreme parallelism.

Purpose of the Study:

• To design and evaluate a storage system that overcomes I/O bottlenecks.
• To achieve high Input/Output Operations Per Second (IOPS) using a user-space file abstraction for SSD arrays.
• To minimize CPU overhead and lock-contention in NUMA systems through redesigned page caching.

Main Methods:

• Developed a user-space file abstraction to refactor I/O scheduling and placement for parallelism.
• Implemented a set-associative, parallel page cache within the user space.
• Redesigned page caching to address NUMA architecture challenges.
• Evaluated the system on a 32-core NUMA machine.

Main Results:

• The storage system achieved over one million IOPS, specifically 1.23 million 512-byte read IOPS.
• The user-space page cache demonstrated scalable IOPS comparable to Linux asynchronous I/O (AIO).
• User-perceived I/O performance increased linearly with cache hit rates.
• The parallel cache achieved hit rates matching the global Linux page cache under real workloads.

Conclusions:

• The proposed storage system effectively removes I/O bottlenecks, enabling high-performance data access.
• User-space file abstraction and parallel page caching are key to achieving extreme parallelism and performance on SSD arrays.
• The redesigned page cache efficiently handles NUMA architectures, reducing overhead and improving scalability.