AP-IO: asynchronous pipeline I/O for hiding periodic output cost in CFD simulation

Area of Science:

• Scientific Computing
• High-Performance Computing
• Computational Fluid Dynamics

Background:

• Periodic snapshot output in CFD simulations significantly impacts performance.
• Existing asynchronous I/O methods have limitations in hiding write overhead.
• Massively parallel CFD simulations are particularly sensitive to I/O bottlenecks.

Purpose of the Study:

• To develop an optimization scheme for periodic snapshot output in CFD simulations.
• To improve the performance of CFD applications by minimizing I/O overhead.
• To provide a user-friendly implementation for CFD practitioners.

Main Methods:

• Developed an asynchronous pipeline I/O (AP-IO) scheme tailored for CFD.
• Implemented AP-IO within the OpenFOAM framework.
• Utilized dedicated background I/O processes/threads for pipelined file writes.
• Evaluated performance on the Tianhe-2 supercomputer.

Main Results:

• AP-IO effectively hides write overhead by overlapping I/O with computation.
• Achieved significant performance improvements for periodic snapshot output.
• Demonstrated substantial reductions in total execution time, up to 40% for massively parallel simulations.
• The AP-IO scheme is user-friendly for CFD applications.

Conclusions:

• AP-IO is an effective optimization strategy for CFD snapshotting.
• The proposed method significantly enhances the performance of large-scale CFD simulations.
• AP-IO offers a practical solution to I/O performance issues in scientific computing.